TR-15 

ADAPTIVE  LINEAR  ESTIMATION  ALGORITHMS 
APPLIED  TO  SPECTRAL  LINE  ENHANCEMENT 


by 

Stephen  D.  Hoffman 
Department  of  Electrical  Engineering 


Prepared  under: 

Office  of  Naval  Research  CCode  222) 
Conti act  No.  N000  14-75-C-0191 


"This  document  has  been  approved  for  public  release 
and  sale;  its  distribution  is  unlimited." 


DUKE  UNIVERSITY 


ADAPTIVE  SIGNAL  DETECTION  LABORATORY 


Department  of  Electrical  Engineering 
School  of  Engineering 


I Technical  fit 


ADAPTIVE  JJNEAR  ESTIMATION  ALGORITHMS 

APPLIED  TO  SPECTRAL  LINE  ENHANCEMENT 
•*  -r 


Stephen  D. /Huffman 


* SNtltt 


Approved 


NOLTE 


Principal  Investigator 


Prepared  under:  Office  of  Navj 
Contract -No.fi 


"This  document  has  been  approved  for  public  release  and  sale;  it 
distribution  is  unlimited." 


This  report  was  also  a dissertation 
submitted  in  partial  fulfillment  of  the 
requirements  for  the  Doctor  of  Philosophy 
degree  in  the  Department  of  Electrical 
Engineering  in  the  Graduate  School  of  Arts 
and  Sciences  of  Duke  University.  1978. 


Copyright  by 
Stephen  Doyle  Huffman 
1978 


I 


ABSTRACT 

The  estimation  of  a signal  in  the  presence  of  an  additive 
noise  process  is  a basic  problem  in  communications  theory. 

When  the  processor  chosen  to  carry  out  the  estimation  task  is 
a linear  digital  filter  with  a finite  unit  sample  response,  the 
design  problem  is  to  determine  the  unit  sample  response  which 
will  produce  the  best  possible  estimate  of  the  signal  according 
to  some  criterion.  If  the  criterion  chosen  as  a measure  of 
optimality  is  the  mean-square  error  of  estimation  and  if  the  sta- 
tistics of  the  signal  and  noise  are  known,  the  optimal  unit 
sample  response  can  be  found  by  the  solution  of  a system  of 
linear  equations.  The  resulting  processor  is,  then,  an  example 
of  the  classical  Wiener  filter. 

When  the  statistics  of  the  signal  and  noise  are  not  known 
a priori,  the  unit  sample  response  of  the  Wiener  filter  cannot 
be  determined  and  some  method  for  approximating  the  optimal 
filter  is  required.  One  approach  to  this  problem  is  to  use  an 
adaptive  processor  which  attempts  to  "learn"  the  characteristics 
of  the  signal  and  the  noise  and  to  use  this  information  to 
obtain  an  approximation  to  the  optimal  filter.  The  LMS  Gradient 
Algorithm  is  a widely  used  method  for  the  solution  of  the 
adaptive  linear  estimation  problem  which  has  the  advantages  of 
being  relatively  simple  to  implement  and  of  being  well  suited  for 
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real-time  processing  in  many  applications.  However,  since  this 
algorithm  is  iterative  in  nature,  the  convergence  properties  of 
the  method  must  be  considered  in  an  evaluation  of  performance. 

In  this  dissertation,  two  adaptive  algorithms  based  on 
fixed-point  iteration  methods  and  using  direct  estimates  of  the 
required  statistics  are  proposed.  These  algorithms  are  slightly 
more  complex  than  the  LMS  Gradient  Algorithm  but  have  the  ad- 
vantage that  the  amount  of  computational  effort  required  during 
each  sample  period  can  be  varied.  Thus,  trade-offs  can  be  made 
between  the  number  of  operations  required  per  sample  and  the 
rate  of  convergence  of  the  algorithm. 

The  performance  characteristics  of  the  LMS  Gradient  Al- 
gorithm, two  Adaptive  Fixed-Point  Iteration  Algorithms  and  of 
a non-iterative  method  based  on  Levinson ' s Algorithm  are  con- 
sidered for  the  case  where  an  adaptive  algorithm  is  used  to 
determine  the  unit  sample  response  for  a system  which  attempts 
to  discriminate  between  the  signal  and  noise  processes  on  the 
basis  of  bandwidth.  Such  a system  is  often  referred  to  as  a 
Spectral  Line  Enhancer.  Theoretical  bounds  on  the  mean-square 
error  as  a function  of  the  time  index  n are  derived  for  each  of 
the  three  iterative  methods . A comparison  of  these  bounds  is 
made  for  the  case  where  the  input  data  to  the  Spectral  Line 
Enhancer  is  composed  of  a single  sinusoid  of  random  phase  in 
the  presence  of  an  additive  autoreqressive  noise  sequence.  The 
results  of  extensive  computer  simulations  of.  the  adaptive 
algorithm  considered  are  used  to  determine  the  usefulness  of  the 
theoretical  bounds  and  to  make  comparisons  of  the  performance  of 
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the  four  methods.  The  results  of  the  computer  simulations 
indicate  that  the  theoretical  bounds  on  the  mean-square  error  are 
close  to  the  actual  mean-square  error  produced  by  the  filters 
determined  by  the  three  iterative  algorithms  for  the  case  con- 
sidered. In  addition,  a comparison  of  the  theoretical  bounds  and 
the  computer  simulation  results  indicates  that  the  Adaptive 
Fixed-Point  Iteration  Method  using  the  Successive  Over-Relaxation  j 
Algorithm  can  yield  significantly  better  performance  than  the 
LMS  Gradient  Algorithm  with  an  increase  in  computational  complex- 
ity which  would  not  be  prohibitive  in  many  real-time  applications. 
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ADAPTIVE  LINEAR  ESTIMATION  ALGORITHMS 
APPLIED  TO  SPECTRAL  LINE  ENHANCEMENT 


CHAPTER  I 

ADAPTIVE  LINEAR  ESTIMATION  AND  SPECTRAL  LINE  ENHANCEMENT 

A basic  problem  in  communications  theory  is  the  estimation 
of  a signal  S(t)  in  the  presence  of  an  additive  noise  process 
S(t).  The  available  information  about  the  signal  is  then  in  the 
form  of  the  received  data  given  by  the  sum: 

X(t)  = S(t)  + N ( t)  . (1.1) 

The  signal  processing  system  which  is  assigned  the  task  of  esti- 
mating  S(t)  must,  given  the  input  signal  X(t),  produce  an  output 
which  in  some  sense  approximates  S(t).  Thus,  the  basic  estima- 
tion problem  is  illustrated  in  Fig.  1.1  where  the  signal  proces- 
sor is  represented  by  a two-part  "black  box". 

1.1  Discrete  Time  Estimation 

The  advances  in  the  development  of  high-speed  digital 
processors  have  led  to  their  widespread  use  in  the  area  of  signal 
processing.  Within  the  discrete-time  framework  of  a digital  sys- 
tem, the  basic  estimation  problem  of  Fig.  1.1  may  be  reformulated 
as  the  estimation  of  a sample  sequence  S(n)  in  the  presence  of 
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an  additive  noise  sequence  N(n).  The  received  information  is 
then  in  the  form  of  the  data  sequence : 

X (n)  = S (n)  + N (n)  (1.2) 

where  it  is  implied  that  if  T is  the  sampling  period: 

S (n)  - S(nT) 

N (n)  = N(nT)  (1.3) 

X (n)  = X(nT) 

This  thesis,  then,  is  concerned  with  the  discrete-time  estimation 
problem  illustrated  in  Figure  1.2  where  the  Digital  Signal  Pro- 
cessor block  receives  the  input  sequence  X(n)  and  produces  an 
output  sequence  S(n)  which  in  some  sense  approximates  the  se- 
quence S (n) . 

1.2  Linear  Estimation 

If  ther  Digital  Signal  Processor  in  Fig.  1.2  produces  the 

A 

output  S(n)  by  applying  a linear  transformation  to  the  data 
sequence  X(n)  then  the  estimation  is  said  to  be  linear.  In  the 
case  considered  here,  it  is  assumed  that  the  processor  is  in  the 
form  of  a causal,  linear  digital  filter  with  a finite  unit  sample 
response  h(n).  That  is,  h(n)  may  take  on  non-zero  values  only 
over  the  finite  interval: 


Figure  1.2  Discrete  Time  Estimation  Problem 
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O£n1£n£n2<00 

and  the  signal  estimate  sequence  may  be  expressed  as  the  linear 
convolution  of  the  data  X(n)  with  the  unit  sample  response  h(n). 
That  is : 


S (n) 


h(i> 


X (n-i) 


(1.4) 


The  estimate  S(n)  can  be  represented  in  a more  compact  form  by 
introducing  the  following  vector  notation: 


h=  [hCn^,  h (n1+l) . . .h  (n2)  ] 


(1.5) 


Xfn-n^)  = (X(n-n^) ,X(n-n^-l) , . . .X(n-n2) ] 


(1.6) 


With  this  notation,  S(n)  may  be  expressed  as  the  scalar  product: 


S (n)  » X^n-n^h  * hT  Xfn-n^  . (1.7) 

It  is  further  assumed  here  that  the  filter  is  implemented 
in  such  a fashion  so  as  to  allow  h(n)  to  be  specified  for  each 
value  of  n independently.  Figure  1.3  gives  a conceptual  block 
diagram  of  the  digital  estimation  process  considered  in  this  the 
sis.  Such  a system  is  often  referred  to  as  a transversal  filter 
or  a weighted  tap-delay  line. 
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1.3  Mean- Square  Error  and  Optimum  Estimation 

In  order  to  determine  how  closely  the  signal  estimate 

| 

S(n)  approximates  the  true  signal  sequence  S(n),  some  measure  of 
the  accuracy  of  the  estimate  is  needed.  The  measure  most  common- 
ly used  is  the  mean-square  error  (MSE)  given  by: 

E [e2 (n) ] = E [ (S (n) -S  (n) ) 2 ] (1.8) 

where  the  notation  E [ * ] denotes  the  expected  value  operator. 

/\ 

Within  a class  of  estimators,  an  estimate  S(n)  is  said  to  be 

2 

optimum  in  the  sense  of  minimum  MSE  in  E Le  (n) ] for  the  estimate 

2 

is  less  than  or  equal  to  E [e  (n) ] for  any  other  estimates  within 
the  class. 

1.4  Solution  for  Minimum  Mean-Square  Error 

If  the  class  of  estimators  considered  is  that  given  by 
(1.4)  or  (1.7),  then  the  problem  to  be  solved  is  to  find  the  unit 
sample  response  h(n)  of  the  filter  so  that  (1.8)  is  minimized. 
While  the  solution  of  this  problem  is  well  known  and  results  in 
the  discrete  form  of  the  Wiener  Filter  [1],  the  development  in 
this  section  is  included  for  completeness  arid  will  serve  as  a 
starting  point  for  the  discussion  of  real-time  adaptive  linear 
estimation  algorithms. 

: 


I 


; 
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For  each  value  of  n,  the  signal  sequence  sample  S (n)  is  to 
be  estimated  by  a linear  combination  ~f  N = (n2~n1+l)  samples  of 
the  data  sequence  X(n)  as  seen  in  (1.4).  The  N sample-values  of 
the  unit  sample  response  h(n)  are  to  be  chosen  so  as  to  minimize 
the  resulting  mean-square  error  given  by  (1.8).  This  optimum 
solution  may  be  obtained  by  applying  the  orthogonality  principle 
which  states  that  the  constants  h(n)  which  minimize  (1.8)  must  be 
such  that  the  estimation  error  is  orthogonal  to  the  data  [2] . 

That  is  the  vector  identity: 

El  (S(n)-S(n) ) X(n-nx)]  = 0 (1.9) 

must  be  satisfied.  Substituting  (1.7)  into  (1.9)  yields: 

i 

E[ (S(n)-XT(n-n1)h)X(n-n1)]  =0 


E[S(n)X(n-n1)  - X (n-n^  XT  (n-n^  h]  = 0 . 

By  the  linearity  of  the  expected  value  operator  [ 3] : 

E[S(n)X(n-n1)  ] - E [X (n-n^  XT (n-n^  ]h  = 0 
or: 


(1.10) 


E [X  (n-n1 ) X-1  (n-n1 ) ] h = E [S  (n)  X(n-n^)  ] . 

With  the  following  definitions,  (1.10)  can  be  stated  in  the 
familiar  matrix  form: 

T 

4>  (n-n.  ,n-n.  )=E  [x(n-n. ) X (n-n. ) 1 = Input  Autocorrelation 

* 1 Matrix  (NxN) 

Rsx(n,n-n.)=E [S (n)X(n-n.) ] = Cross-correlation  vector  between 

signal  and  data  sequences  (Nxl) 

For  convenience,  the  time  index  will  be  dropped  in  the  above  de- 
finitions and  unless  specifically  stated  otherwise  it  will  be 
implied  that: 


$xx  = ^XX (n-n^ ,n-n^)  (1.11a) 

-SX  = -SX*n,n-nl^  * (1.12a) 

With  the  above  definitions,  (1.10)  becomes: 


txxh-SsX  . (1.13) 

If  the  above  system  has  a unique  solution  then  4>xx  is  non- 
singular and  thus  has  an  inverse.  Then  the  optimum  unit  sample 


(1.14) 


It  remains  to  be  shown  that  $xx  is  invertable , or  equiva- 
lently that  the  mean-square  error  given  by  (1.8)  has  a unique 
minimum  point.  If  (1.8)  is  expanded  using  the  linearity  property 
of  the  expected  value  operator,  the  following  equalities  are 
obtained: 

Efe2  (n)  ] = E[S2(n)-2S(n)S(n)+S2(n)] 

E[  e2  (n)  ] = E[S2(n)]-2E[S(n)S(n)]+E[S2(n)]  . (1.15) 

Substitution  of  (1.7)  into  (1.15)  yields: 

E [e2  (n)  ] = E[S2(n)  ]-2hTE[S(n)X  (n-n^  ]+hTE  [X  (n-n^  XT  (n-n^  ]h 

and  by  applying  the  definitions  (1.11a)  and  (1.12a)  the  following 
expression  for  the  mean-square  error  is  obtained: 

E[e2 (n) ] = E[S2 (n)] -2hTRsx+hT$xxh  . (1.16) 

The  above  expression  for  the  mean-square  error  is  quadratic  in 
the  terms  of  the  unit  sample  response  h(n)  and  can  be  shown  to 
have  a unique  minimum  point  if  the  following  necessary  and  suffi- 
cient conditions  are  met  [4 ] : 

2 

1.  The  gradient  of  E [e  (n) ] with  respect  to  h is  equal 
to  zero. 

2 

2.  The  Hessian  matrix  of  Efe  (n) ] with  respect  to  h is 
positive  definite. 
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Applying  these  conditions  to  (1.16)  yields  the  following  require- 
ments : 

1.  2^xX^”-SX^  * — * 

2.  YT<!>xxY  > 0 v Y e RN  (i.e.  $xx  is  positive  definite). 

If  is  positive  definite,  it  is  invertable  and  thus  h given  by 

(1.14)  will  satisfy  condition  (1).  Therefore  it  remains  to  be 
shown  only  that  4>xx  is  positive  definite. 

T 

Theorem  1.1.  The  matrix  $xx  = E[X(n-n1)X  (n-n^) ] is  positive 
definite  provided  that  P[X(n-n^)]  is  not  a degenerate  distribu- 
tion [5]. 

Proof;  Let  Y be  an  arbitrary  N-vector  and  define  the  scalar 
random  variable; 

a * YTX (n-nL)  = XT(n-n1)Y  (1.17) 

consider; 

VAR[  a]  - E[a2]  - (E[aJ  )2 
VARta]  = E[YTX(n-n1)XT(n-n1)  Y] -(E[a]  ) 2 
VAR[ a]  = YTE[X(n-n1)XT(n-n1)]  Y-(E[a]  )2 
VARta]  = YT$XXY-  (E[aj  )2 

Now: 


VARta]  > 0 . 


VAR  [a]  = 0 if  and  only  if  a=alf  a constant,  with  P[a»a^]*l. 

If  this  is  true,  then  from  (1.17)  PfXtn-n^)]  must  equal  zero 
everywhere  except  on  the  linear  variety  given  by: 

al  31  ^TX(n-n1) 

N 

which  is  an  (N-l)  dimensional  subspace  of  R . Thus,  PtXfn-n^)] 

is  degenerate  on  N-dimensional  space.  Therefore,  if  P[X(n-n^)] 

. . 2 
is  not  degenerate,  and  since  (E[a])  >0 : 

T N 

- XX->0  v -£R 

and  thus,  it  has  been  shown  that  $xx  is  positive  definite. 

Since  the  necessary  and  sufficient  conditions  are  satis- 
fied, as  has  been  shown,  the  mean-square  error  (1.8)  has  a 
unique  minimum  point  h*.  Therefore  4>  is  invertable  and  the 
optimum  unit  sample  response  is  given  by  (1.14). 

1.5  Application  of  the  Optimal  Filter  and  Adaptive  Filtering 

It  has  been  shown  that,  the  optimal  filter  characterized 
by  the  unit  sample  response  h*(n)  given  by  (1.14)  will  yield  the 
minimum  mean-square  error  estimate  of  S(n)  for  the  class  of 
filters  given  by  (1.4)  when  applied  to  the  data  sequence  X(n). 
However,  the  compact  closed-form  solution  of  (1.14)  is  mislead- 
ing when  the  difficulty  of  implementing  such  a filter  is  consi- 
dered. Specifically,  (1.14)  implicitly  assumes  that  the  auto- 
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correlation  matrix  <^x  and  the  cross-correlation  vector  RgX 
are  known  a priori  so  that  the  optimum  unit  sample  response  can 
be  determined  and  the  optimal  filter  implemented.  However,  in 
practice,  the  autocorrelation  matrix  and  the  cross-correlation 
vector  may  not  be  known  exactly.  In  this  case,  a filter  cannot 
be  determined  by  the  use  of  (1.14)  and  another  approach  to  the 
problem  must  be  chosen.  One  possible  method  which  could  be 
applied  in  this  situation  would  be  to  determine  a FIR  digital 
filter  which  would  in  some  sense  approximate  the  optimal  filter. 

One  approach  to  obtaining  a filter  is  to  estimate  the 
input  autocorrelation  matrix  <ixx  and  the  cross-correlation  vector 
Rsx  from  the  input  data  sequence  X(n)  and  to  use  these  estimates 
to  find  a unit  sample  response: 

- = *XX  ^SX  (1.18) 


which  can  be  considered  as  an.  example  of  an  adaptive  filter. 
That  is,  the  unit  sample  response  of  the  filter  is  determined 
from  estimates  of  the  input  statistics  based  on  the  data  which 


is  to  be  filtered.  If  the  estimates  4>xx  and  RgX  and  the  unit 
sample  response  h are  updated  continually  by  some  algorithm  based 
on  the  old  estimates  and  the  newly  received  data,  then  the  adapta-j- 
tion  may  be  considered  to  be  recursive.  The  remainder  of  this 
thesis  is  concerned  with  algorithms  which  attempt  to  approximate 
the  minimum  mean-square  error  criterion  by  the  recursive  adapta- 
tion of  the  filter  unit  sample  response  based  on  estimates  of  the 

input  statistics  when  the  autocorrelation  matrix  and  cross- 
correlation vector  are  not  known  a priori. 


Examination  of  (1.18)  reveals  that  estimates  of  the  auto- 
correlation matrix  of  the  input  data  and  the  cross-correlation 
vector  of  the  signal  with  the  input  data  are  needed  for  the 
adaptive  processing.  Since  no  a priori  knowledge  of  the  signal 
is  assumed,  a form  of  (1.13)  in  which  only  correlations  of  the 
data  occur  is  desirable.  Substituting  (1.2)  into  (1.13)  yields: 


ijjjjh  = E [ (X  (n)  -N  (n) ) X (n-n^)  ] 


By  the  linearity  of  the  expected  value  operator 


$xxh  = E[X(n)X(n-n1)  ]-E[N(n)X(n-n1)  ] 


(1.19) 


Now,  since  X(n-n^)-S  (n-n^  +N  (n-n^  , (1.19)  becomes 


$xxh  * ElX(n)X(n-n1)]-E[N(n)S(n-n1)] -E[N(n)N(n-n1)]  . (1.20) 


At  this  point,  two  simplifying  assumptions  will  be  made.  First, 
it  is  assumed  that  the  signal  sequence  S(n)  and  the  noise  pro- 
cess N(n)  are  orthogonal.  Secondly,  the  assumption  is  made 
that  the  delay  n1  in  the  unit  sample  response  is  such  that  noise 
samples  which  are  separated  by  n^  or  more  samples  in  the  sequence 
are  also  orthogonal.  These  two  assumptions  imply  respectively 
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1.  E [N  (n)  S (n-n..  ) ] = 

0 

(1.21) 

— 1 

2.  E[N(n)N(n-n1) ] = 

0 

(1.22) 

Thus,  (1.20)  becomes: 

$XX-  = E[X(n)X(n-nL)] 

. 

(1.23) 

With  the  following  definition,  the  alternate  form  of  the  optimal 
solution  given  assumptions  (1.21)  and  (1.22)  is  obtained. 

It...  (n, n-n.)  * E [X (n) X (n-n. ) 1 = Autocorrelation  vector  of 

1 the  input  data  (NX1) . (1.24) 

Again,  the  time  index  will  be  dropped  for  convenience  and  it 
will  be  implied,  unless  specifically  stated  otherwise  that: 

* R^fn^-l)  . (1.24a) 

With  the  above  definition,  (1.23)  becomes: 

^XX~  * ^XX  * (1-25) 

And  the  optimal  unit  sample  response  is  given  by: 

-*  * $XX  -XX 


(1.26) 
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The  above  form  for  the  optimal  solution  involves  only  correla- 
tions of  the  input  data.  Thus,  applying  assumptions  (1.21)  and 
(1.22)  has  eliminated  the  need  for  knowledge  of  the  cross- 
correlation between  the  signal  and  the  data.  The  remaining  terms 
can  be  more  easily  estimated  from  the  data. 

1.6  Separation  of  Signals  Based  on  Bandwidth 

Assumptions  (1.21)  and  (1.22)  allow  the  optimum  unit 
sample  response  to  be  written  in  the  more  tractable  form  of 
(1.26).  However,  these  assumptions  and  the  inclusion  of  the 
delay  n1  in  the  unit  sample  response  place  restrictions  on  the 
conditions  under  which  (1.26)  will  yield  a useful  solution.  A 
consideration  of  these  restrictions  leads  to  am  importamt  proper- 
ty of  the  estimator  which  is  not  immediately  evident  in  (1.26) 
but  which  must  be  considered  if  useful  results  are  to  be  ob- 
tained. 

If  N(n)  or  S(n)  is  a zero-mean  process,  the  orthogonality 
condition  of  (1.21)  is  equivalent  to  the  requirement  that  the 
signal  and  the  noise  be  uncorrelated.  Similarly,  if  N(n)  has 
zero  mean,  (1.22)  will  be  satisfied  if  noise  samples  which  are 
separated  by  n1  or  more  samples  are  uncorrelated.  In  this  case,  ; 
the  minimum  value  of  n^  needed  to  satisfy  (1.22)  is  often  re- 
ferred to  as  the  required  "decorrelation  delay"  [6].  Since 
S(n)  and  N(n)  are  orthogonal,  (1.26)  can  be  rewritten  as: 

h*  = *xX(-SS+£nn)  = «^(E[S(n)S(n-n1)]+E[lI(n)N(n-n1)]).  (1.27) 


Applying  (1.22)  to  (1.27)  yields: 

h*  = *J^E[S(n)S(n-n1)]  . (1.28) 

Examination  of  (1.28)  reveals  that  if  n^  is  also  sufficient  to 
" decorrelate"  the  signal,  then  (1.26)  yields  only  the  trivial 
solution : 

h*  = 0 

Thus,  the  solution  given  by  (1.26)  is  only  useful  if  the  signal 
is  more  correlated,  or  coherent,  than  the  noise.  That  is,  the 
signal  autocorrelation  function  must  be  significantly  non-zero 
for  larger  delays  than  is  the  autocorrelation  function  of  the 
noise.  Since  the  power  spectral  density  and  the  autocorrelation 
function  are  related  by  the  Fourier  Transform,  bandwidth  and 
length  of  the  autocorrelation  delay  are  inversely  related.  Thus, 
a process  with  a relatively  narrow  bandwidth  will  have  an  auto- 
correlation function  which  is  significantly  non-zero  for  larger 
delays  than  the  autocorrelation  function  of  a wideband  signal. 
Therefore,  the  solution  given  by  (1.26)  is  only  useful  for  the 
case  where  the  process  S(n)  and  N(n)  have  significantly  different 
bandwidths. 

Consideration  of  assumption  (1.22)  and  expression  (1.28) 
reveals  that  the  signal  estimate  given  by: 


S (n)  = h*TX (n-n.  ) 


(1.29) 
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where  h*  is  given  by  (1.26)  or  (1.28),  is  only  valid  if  a value 
of  n1  cam  be  found  which  will  decorrelate  the  noise  without 
decorrelating  the  signal.  Thus,  from  the  preceeding  discussion 
it  is  apparent  that  S(n)  must  have  a relatively  narrow  bandwidth 
compared  to  that  of  N(n).  If  this  is  the  case,  an  appropriate 
value  of  n^  can  be  found  and  the  signal  can  be  estimated  using 
the  system  shown  in  Fig.  1.4  where  the  unit  sample  response  of 
the  causal  linear  digital  filter  is  given  by  (1.26). 

Conversely,  if  the  signal  has  a wide  bandwidth  compared  to 
that  of  N(n),  a value  of  n1  which  decorrelates  the  noise  and  not 
the  signal  cannot  be  found.  In  this  case,  one  approach  which  has 
been  suggested  is  to  estimate  the  noise  and  to  form  the  signal 
estimate  by  a noise-cancelling  procedure  [7] :* 

S(n)  = X(n)-N(n)  (1.30) 

where  the  noise  estimate  is  given  by: 

N(n)  = h^X(n-n1)  = X^n-n^l^  . (1.31) 

It  has  been  assumed  in  the  above  expression  that  n1  has  been 
chosen  to  decorrelate  the  wideband  signal  but  not  the  narrow- 
band  noise.  That  is,  it  is  assumed  that: 

E[S(n)S(n-n1)]  = 0 (1.32) 
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and  also  that  S(n)  and  N(n)  are  orthogonal,  or  that  (1.21) 
holds.  Figure  1.5  gives  a block  diagram  of  a noise-cancelling 
system  for  the  estimation  of  a wideband  signal  in  the  presence 
of  narrowband  noise. 

The  unit  sample  response  hN(n)  must  be  determined  so  that 

A 

S(n)  given  by  (1.30)  is  the  minimum  mean-square  estimate  of  S(n) 
Applying  the  orthogonality  principle  to  the  estimate  in  (1.30) 
yields : 

E[X(n-n1)  (S(n)-(X(n)-N(n)  ) )]  =0 
E[X(n-nL)  (S(n)-(X(n)-XT(n-n1)hN) )]  =0  (1.33) 

Erxfn-n^Sfn)]  -E[ X (n-n2 ) X (n)  ] -E[ X (n-n^ XT (n-n^ ] = 0 . ‘ 

Applying  assumptions  (1.21)  and  (1.22)  to  the  above  expression 
gives : 


E[  Xfn-n^X^n-n^]  h^  = E[  X (n)  X (n-n^ ] 

Finally,  applying  the  definitions  (1.6)  and  (1.24)  yields: 


$XX^  ^XX  * 

The  optimum  unit  sample  response  is  then  given  by: 

[ = $XX^XX 


Comparison  of  (1.34)  and  (1.26)  reveals  that  the  same  optimum 
unit  sample  response  is  required  regardless  of  whether  the  sig- 
nal or  the  noise  is  relatively  narrowband.  That  is:  ‘ 

h*  = h*  . (1.35) 

I 

In  the  case  of  relatively  narrowband  noise,  the  estimate 

N(n)  given  by  (1.31)  with  h^h*  is  the  minimum  mean-square  error 

estimate  of  the  noise.  This  can  be  verified  by  considering  (1.33)! 

for  which  h*  is  the  solution: 

— N 

E[X(n-n1)  (S(n)-(X(n)-N(n)))l  =0 
E[X(n-n^)  (N(n)-N(n))]  = 0 
E[X(n-n1)  (N(n)-N(n)  )]  =0 

Now,  (1.36)  is  simply  the  statement  of  the  orthogonality  princi- 

A A 

pie  applied  to  the  noise  estimate  N(n).  Therefore,  N(n)  is  the 

I 

minimum  MSE  estimate  of  N(n)  for  the  class  of  estimators  con- 
sidered. Thus,  it  is  obvious  that  when  a causal,  linear  digital 
filter  with  a unit-sample  response  given  by  (1.26)  is  applied  to 
a data  sequence  composed  of  relatively  wideband  and  relatively 
narrowband  components,  the  filter  output  given  by: 

Xjjg  (n)  = h^Xfn-n^ 


(1.36) 


(1.37) 
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is  the  minimum  MSE  estimate  of  the  narrowband  components  of  the 
data  while: 

X^n)  = X(n)-XNB(n)  (1.38) 

is  the  minimum  MSE  estimate  of  the  wideband  components  of  the 
data.  Expressions  (1.37)  and  (1.38)  hold  provided  that  the 
decorrelation  delay  n^  has  been  chosen  so  as  to  decorrelate  the 
wideband  components  but  not  the  narrowband  components.  Fig.  1.6 
is  a block  diagram  of  a system  which  combines  the  estimation  of 
the  narrowband  and  wideband  components  of  the  data.  This  system, 
which  attempts  to  discriminate  between  two  processes  on  the 

basis  of  bandwidth  is  an  example  of  a Spectral  Line  Enhancer  [8]. 

• 

1.7  Algorithms  for  Approximation  of  the  Optimal  Solution 

The  Spectral  Line  Enhancer  of  Fig.  1.6  may  be  viewed  as 
a generalized  estimator  for  signal  and  noise  processes  which 
differ  in  bandwidth  given  that  the  restrictions  discussed 
above  are  considered.  The  remaining  task,  then,  is  to  obtain  am 
approximation  h(n)  to  h*(n).  Historically,  there  are  two  basic 
methods  for  obtaining  such  an  approximation.  The  first  involves 
the  collection  of  a block  of  input  data,  the  estimation  of  $xx 
and  Rxx  from  this  block  of  data  and  the  solution  of  the  linear 
system  of  equations  (1.25)  to  obtain  h(n)  which  can  then  be 
used  to  filter  the  block  of  data.  Methods  proposed  for  the 
solution  of  (1.25)  include  Triangularization  [9],  Gaussian 


Narrowband 

Process 


Wideband 

Process 


Figure  1.6  Spectral  Line  Enhancer. 


26 


Elimination  [10],  Steepest  Descent  [11],  Conjugate  Gradient  [11J , 
and  Levinson-Robinson  [12],  The  second  basic  method  is  the  use 
of  an  adaptive  recursive  algorithm  such  as  the  LMS  Gradient 
Method  [13].  Accumulation  of  data  and  direct  solution  of  (1.25) 
may  give  a better  approximation  to  h(n)  than  an  adaptive  recursive 
algorithm.  However,  such  a method  requires  considerable  computa- 
tional effort  compared  to  that  needed  by  an  algorithm  such  as 
the  LMS  Gradient  Method.  The  approach  taken  in  this  thesis 
lies  somewhere  between  the  two  basic  methods  both  in  terms  of 
computational  effort  and  performance.  In  this  new  approach,  the 
received  data  samples  are  used  to  update  the  estimates  of  $xx 
and  R^  as  in  the  first  approach.  However,  rather  than  solve 
(1.25)  directly,  one  step  of  an  iterative  method  such  as  Jacobi 
Relaxation  or  Successive  Over- Relaxation  is  used  to  update  h(n) 
based  on  the  previous  approximation  and  the  updated  estimates  of 
$xx  and  RjQf*  The  conceptual  flow  diagram  of  Fig.  1.7  illustrates 
the  algorithm  for  the  adaptation  of  the  unit-sample  response. 

The  remainder  of  this  thesis  is  concerned  with  the  development 
of  Adaptive  Fixed-Point  Iteration  Algorithms  based  on  Jacobi 
Relaxation  and  the  Successive  Over-Relaxation  methods  and  with 
comparisons  between  these  methods  and  the  LMS  Gradient  Algorithm. 
The  convergence  properties  of  the  three  methods  are  considered 
and  theoretical  bounds  on  their  rates  of  convergence  are  derived. 
In  addition,  a non-iterative  approach  using  Levinson's  Algorithm  j 
is  also  considered  in  which  estimates  of  $xx  and  are  updated 
at  each  sample  time  n,  but  the  unit-sample  response  is  updated 


2 


Figure  1.7  Conceptual  Flow  Diagram  for 

Adaptive  Fixed-Point  Iteration. 


CHAPTER  II 


THE  LMS  GRADIENT  ALGORITHM 


The  LMS  Gradient  Method  proposed  by  Widrow  [13]  is  a 
widely  used  adaptive  recursive  algorithm  for  the  approximate 
solution  of  the  discrete  Wiener-Hopf  equation.  The  LMS  Gradient 
Algorithm  is  based  on  the  Steepest  Descent  Method  for  the 
minimization  of  a non-linear  function  of  several  variables. 

The  applicability  of  the  method  of  Steepest  Descent  cam  be  seen 
by  considering  the  mean-square  error  as  a function  of  the  terms 
of  the  unit-sample  response,  h(n).  In  Chapter  I,  it  was  shown 
that  the  MSE  was  a quadratic  function  of  the  sample  values  of 
the  unit-sample  response.  Specifically,  it  was  shown  that: 


E[  e2  (n)]  = E[S2(n)] -2hTRsx+hT$xxh 


(2.1) 


where  the  following  definitions  apply: 

h — t h (nx) , h (n1+l)  . . .h(n2)]  T (2.2) 

RgX  = E(  SfnUtn-n^l  (2.3) 

$xx  = E [X (n-nx) XT (n-nx) ] . 


(2.4) 


It  was  also  shown  in  Chapter  I that  4>xx  was  positive  definite  and 


therefore  that  E[e  (n) ] has  a unique  minimum  point  h*.  Thus, 


since  E[e  (n)]  is  quadratic  and  has  a unique  minimum  it  is 


strictly  convex.  Therefore',  given  any  two  N- vectors  h^  and 


^m+1 ' lf 


E[e2  (n)  IhJ  > E [e2  (n)  Ih^] 


(2.5) 


then: 


(hn-h»)T(hm-h*)  > i^-h*) * (h^rfc**  (2-6) 


2 

where  the  notation  E[e  (n)  Ih^]  is  used  to  indicate  the  value  of 


the  MSE  computed  using  (2.1)  with  h replaced  by  h^.  Therefore, 


if  the  MSE  with  h=h  is  less  than  the  MSE  with  h=h  . then  the 

— — m+l  — — m 


distance  from  h^^  to  t*ie  optimal  solution  h*  is  less  than  the 

distance  from  h to  h*.  This  implies  that  if  an  N-vector  h is 
— m — — m 


given  and  a direction  vector  deR  is  chosen  so  that  an  N-vector 

— m 


h_ . , which  is  a distance  u from  h along  d satisfies  the  in- 
equality  given  by  (2.5)  then  hm+1  is  closer  to  h*  than  is  h^. 
Such  a direction  vector  is  referred  to  as  a descent  direction  at 
Now,  the  N-vector  h^^  can  be  expressed  as: 


4*1  * VunA. 


(2.7) 


From  the  preceeding  discussion,  it  is  evident  that  if  Equation 
(2.7)  is  applied  recursively  from  an  arbitrary  starting  point, 
a sequence  of  vectors  h which  converges  uniformly  to  h will  be 
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generated  provided  that  ym  and  d^  are  chosen  properly.  Hence, 
(2.7)  may  be  considered  as  a possible  method  for  the  solution  of 
the  discrete  Wiener-Hopf  equation  given  by  (1.13). 


2.1  Steepest  Descent  Algorithm 

The  recursion  equation  (2.7)  is  incomplete  as  an  algorithm 

unless  methods  for  choosing  d and  y are  specified.  Locally, 

— m m 

a function  of  N variables  decreases  most  rapidly  in  the  direction 
of  the  negative  gradient.  Therefore,  the  Method  of  Steepest 
Descent  uses  the  negative  gradient  evaluated  at  h^  as  the  descent 
direction  d . Thus,  (2.7)  becomes: 


^m+1 


h -y  VE  [e  (n) 
— m m 


h ] 
-mJ 


(2.8) 


where : 

7[e2(n)|hinl  “ 2*xx^m-2— SX  ' 

Thus,  for  the  quadratic  problem  considered,  (2.8)  can  be  ex- 
pressed as: 


^m+1  ^m"2ym  ($XX^V-SX 


(2.9) 


The  parameter  y is  chosen  so  as  to  minimize  the  MSE  in 
m 


That  is,  y_  is 
m 


the  direction  of  the  negative  gradient  at  h^. 
chosen  to  minimize: 


B[.2(n)|hnJ-2Pn(^xhn-Rsx)  . 


(2.10) 


With  y chosen  to  minimize  the  expression  in  (2.10),  Equation 
m 

(2.9)  defines  the  Steepest  Descent  Algorithm  for  this  problem. 

The  Steepest  Descent  Algorithm,  in  the  form  discussed 

above,  is  absolutely  convergent  for  the  problem  considered  due 

to  the  fact  that  the  MSE  function  is  strictly  convex  [4]. 

However,  the  parameter  y must  be  computed  at  each  iteration 

m 

step  adding  to  the  computational  complexity  of  the  algorithm. 

A modification  of  the  Steepest  Descent  Algorithm  uses  a fixed 
value  for  the  parameter  y^  for  all  iteration  steps.  The  recur- 
sion relation  then  becomes : 

^m+1  = ^_2y  ($XX^m"-SX)  * (2.11) 

This  method,  sometimes  referred  to  as  the  Method  of  Simultaneous 
Displacements , is  the  algorithm  upon  which  the  LMS  Gradient  pro- 
cedure is  based.  As  will  be  shown  later,  the  Method  of  Simul- 
taneous Displacements  converges  for  the  quadratic  problem  con- 
sidered provided  that  certain  restrictions  are  placed  on  the 
value  of  the  parameter  y [14] . 

The  algorithm  given  by  (2.11)  defines  a method  which  will 
converge  to  the  unit-sample  response  h(n)  which  minimizes  the 
MSE  given  by  (2.1).  However,  two  problems  with  this  approach 
are  immediately  evident.  First,  as  was  mentioned  previously, 
the  autocorrelation  matrix  $xx  and  the  cross-correlation  vector 
R^y  are  generally  unknown  in  practical  problems.  Thus,  these 
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statistics  must  be  estimated  from  the  input  data  in  order  for 

(2.11)  to  be  a useful  algorithm.  A second  problem  can  be  seen 
by  noting  that  at  each  sample  time  n a number  of  iterations  of 

(2.11)  denoted  by  the  index  m may  be  required  for  the  Simultan- 
eous Displacement  Algorithm  to  converge  to  the  optimal  solution. 
Also,  some  method  for  determining  when  the  algorithm  has  con- 
verged to  within  a desired  distance  from  the  optimal  solution 
must  be  given  to  halt  the  iteration  process.  The  need  for  a 
stopping  criterion  and  the  number  of  iterations  required  may 
prohibit  the  use  of  the  algorithm  given  by  (2.11)  in  real-time 
processing.  This  problem  may  be  avoided  if  it  is  assumed  that 
x(n)  is  a stationary  process  so  that  $xx  and  RgX  are  not  func- 
tions of  the  sample  time  n.  In  this  case,  a possible  approach 
would  be  to  carry  out  a single  step  of  the  iteration  defined  by 

(2.11)  at  each  sample  time  n.  The  algorithm  can  then  be  written 


^n+1  -n"2y ($XX^n"^SX) 


(2.12) 


The  algorithm  given  by  (2.12)  converges  as  the  sample  time  n 
increases  provided  that  certain  restrictions  are  placed  on  the 
value  of  y.  A stopping  criterion  is  not  used  since  in  practice 
4>xx  and  R xx  must  be  replaced  by  estimates  for  adaptive  processing 
and,  hopefully,  these  estimates  will  improve  as  more  data  is 
obtained.  In  addition,  the  continual  updating  of  h(n)  using 

(2.12)  allows  the  algorithm  to  adapt  to  slowly  time-varying 
processes  X(n)  provided  that  the  time  variation  of  $xx  and  Rxx 


3 


can  be  reflected  in  their  estimates. 


The  remaining  feature  needed  to  make  (2.12)  into  an 
adaptive  recursive  algorithm  is  some  method  for  estimating  the 
statistics  of  the  input  process.  Specifically,  knowledge  is 
required  about  the  gradient  of  the  MSE  given  by: 

VE(e2(n)|hn]  - 2($xxhn-Rsx)  • (2.13) 

Substituting  the  definitions  of  4>xx  and  Rgx  given  by  (1.11)  and 
(1.12)  into  (2.13)  yields: 

VEte2(n)|h  1 = 2EtX(n-n.  )XT(n-n. ) lh  -2E(S(n)X(n-n.  ) 1 
— n — 1 — l — n — i 

Now,  since  h^  is  known: 

VE Ce2 (n) I h ] = 2EtX(n-n. )XT(n-n. )h  ]-2E[S(n)X(n-n.) ] . 

' — n — 1 — l — n — l 

But, 

S (n)  ■ XT  (n-n^l^ 

Therefore : 

VE[e2 (n) |h  ] = 2E(X(n-n.)S(n)]-2E[S(n)X(n-n.)l  . (2.14) 
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Assumptions  (1.21)  and  (1.22)  state  respectively  that  the  signal 
sequence  S(n)  and  the  noise  process  N(n)  are  orthogonal  and  that 
noise  samples  separated  by  n^  or  more  samples  are  also  ortho- 
gonal. Noting  that  the  signal  S(n)  may  be  written  as: 


S (n)  = X (n)  -N  (n) 


and  by  applying  assumptions  (1.21)  and  (1.22),  Equation  (2.14) 
can  be  rewritten  as: 


VE[e2(n)|hn]  = 2E[X(n-n1)  S (n)  ] -2E[X(n)  X (n-n^  ] 


VE[e2  (n)  I'h^  = 2E[  (S(n)-X(n)  )X(n-n1)]  . (2.15) 


Therefore,  using  (2.15)  the  iteration  algorithm  (2.12)  can  be 
rewritten  as: 


h . = h +2u  E[ (X(n)-S (n) )X (n-n. ) ] . 

— n+1  — n — 1 


(2.16) 


Thus,  the  problem  of  estimating  the  input  statistics  for  the 
adaptive  algorithm  has  been  reduced  to  the  estimation  of  the 
gradient  of  the  MSE  given  by  (2.15).  It  is  obvious  from  (2.15) 
that  the  estimate  of  the  gradient  given  by: 


VE[e2(n)|hn]  - 2(S(n)-X(n))X(n-n1)  (2.17) 


is  an  unbiased  estimate.  That  is: 

E[VE[e2  (n)  |h  ] ] = E[  2 (S  (n)  -X  (n)  ) X (n-n.  ) ] = VE[e2(n)|hj  . 

Thus,  substituting  the  estimate  of  the  gradient  given  by  (2.17) 
into  (2.16)  yields  the  adaptive  recursive  algorithm: 

h^  - hn+2u(X(n)-S(n)  )X(n-nx)  . (2.18) 

Equation  (2.18)  along  with  the  filter  output  equation  (1.7) 
defines  the  LMS  Gradient  Algorithm  developed  by  Widrow  [ 6]  , [7], 
[13]  . 

It  is  obvious  from  (2.18)  that  the  LMS  Gradient  Algorithm 
is  an  attractive  approach  to  real-time  adaptive  estimation  due 
to  its  simplicity.  The  number  of  operations  required  is  on  the 
order  of  the  length  N of  the  unit-sample  response  h(n).  However 
the  simplicity  of  the  algorithm  must  be  weighed  against  its 
performance.  Specifically,  the  convergence  properties  of  the 
LMS  Gradient  Algorithm  must  be  considered  as  well  as  the  number 
of  operations  required  for  its  implementation. 

2.2  Convergence  of  the  LMS  Gradient  Algorithm 

Convergence  properties  are  very  important  to  consider  in 
the  evaluation  of  iterative  algorithms.  Specifically,  it  is 
desirable  to  know  under  what  conditions  the  algorithm  will 
converge  to  the  optimal  solution  h* (n)  and  at  what  rate  the 


convergence  will  occur.  Ideally,  an  algorithm  should  be  ab- 
solutely  convergent.  That  is,  it  should  converge  to  the  optimal 
solution  under  all  conditions  and  with  no  restrictions.  In 
addition  the  convergence  should  be  rapid  so  that  the  optimal 
solution  is  found  after  as  few  iterations  as  possible.  As 
mentioned  previously,  there  are  restrictions  on  the  value  of 
the  parameter  u in  Equation  (2.18)  which  must  not  be  violated  if 
the  algorithm  is  to  converge.  While  already  a part  of  the 
literature  [6] , these  restrictions  are  derived  in  the  next  sec- 

1 

tion  for  completeness.  In  addition,  a bound  on  the  rate  of  con-  | 
vergence  of  the  LMS  Gradient  Algorithm  is  developed  so  that 
comparisons  with  the  Adaptive  Fixed-Point  Iteration  Algorithms 

to  be  introduced  later  can  be  made. 

' 

It  is  important  to  note  at  this  point  that  in  the  dis- 
cussion that  follows  it  is  assumed  that  X(n)  is  a stationary 
process.  While  the  convergence  properties  of  the  LMS  Gradient 
Algorithm  have  been  explored  for  non-stationary  inputs,  closed- 
form  solutions  which  would  allow  comparisons  with  other  algor- 
ithms are  difficult,  if  not  impossible  to  obtain. 

The  LMS  Gradient  Algorithm  as  developed  by  Widrow  [6] , 

[7],  [13],  is  completely  described  by  (2.18)  and  the  filter 
output  equation  (1.7).  These  expressions  are  given  below  in  a 
slightly  different  form  which  will  be  referred  to  as  the  LMS 
Gradient  Algorithm  in  the  remainder  of  this  discussion: 
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(2.19) 

(2.20) 

(2.21) 

Equation  (2.19)  is  identical  to  the  filter  output  equation  (1.7) 
while  equations  (2.20)  and  (2.21)  may  be  combined  to  form  the 
adaptive  recursive  algorithm  given  by  (2.18). 

To  determine  the  restrictions  on  the  value  of  y under 
which  the  LMS  Gradient  Algorithm  is  convergent,  it  is  necessary 
to  consider  the  adaptation  algorithm  given  by  (2.18)  which  is 
repeated  below. 

hn+1  = hji+2y  (X(n)-S  (n)  )X(n-n1)  . (2.18) 

Substituting  (2.19)  into  the  above  expression  yields: 

h . = h +2y  (X(n) -XT(n-n.  )h  )X(n-n. ) 

— n+1  — n — 1 — n — 1 

or: 

h . . = h +2yX (n) X (n-n. ) -2yx (n-n. ) X1 (n-n. ) h 
—n+1  — n — l — l — l — n 

Collecting  terms  on  the  right  hand  side  gives : 

h , . = [I-2yX(n-n. )XT(n-n. ) ]h  +2yX (n) X (n-n. ) . (2.22) 

—n+1  — 1 — 1 — n — 1 


S (n)  = XT  (n-n.  )h 
— i — n 

hn+1  = hii+2ye  (n)X(n-n1) 
e(n)  » X (n)  -S  (n) 


Obviously,  h^^  defined  by  (2.22)  is  a random  vector.  Thus, 
"convergence"  of  ^n+1  to  h*  must  be  interpreted  as  convergence 
in  the  mean.  That  is,  if  the  LMS  Gradient  Algorithm  "converges", 
it  is  in  the  sense  that  h^  is  an  asymptotically  unbiased  esti- 
mator of  the  optimal  unit-sample  response  h*.  Therefore,  taking 
the  expected  value  of  both  sides  of  (2.22)  gives  the  following 
development : 

Elh^]  = E[  (I-2yX(n-nJL)XT(n-n1)  +2yE[X(n)X(n-n1)] 

Elh^]  = Eth^] -2yE[X(n-n1)XT(n-n1)hn] +2yE[X(n)X(n-n1)]  (2.23) 

E[hn+1]  = (I-2yE[X(n-n1)XT(n-n1)  1^]  jEtl^] +2yE[X(n)X(n-n1)] 
where  the  notation: 


E[*  I*] 

indicates  the  conditional  expected  value.  Previous  research 

has  shown  that  if  the  value  of  y is  small,  then  the  effect  of 

T 

the  covariance  between  X(n-n^)X  (n-n^)  and  is  negligible  [13]. 
Under  this  assumption,  (2.23)  can  be  rewritten  as: 

= (I-2uE(X(n-n  )XT(n-n  ))  )E[h  ) +2pE[X(n)X(n-n, )] 
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Substituting  definitions  (1.6)  and  (1.24)  into  the  above  expres- 
sion yields: 


E[W  * (I~2w1XX)E^'  +2u®XX 


(2.24) 


Thus,  under  the  assumption  that  y is  small,  the  convergence 
properties  of  the  LMS  Gradient  Algorithm  can  be  determined  from 
a consideration  of  (2.24).  Equation  (2.24)  is  an  example  of 
fixed-point  iteration  with  the  iteration  matrix: 


B = (I-2y<J>xx) 


Therefore,  the  following  theorem  concerning  convergence  applies. 
A proof  appears  in  Appendix  A. 

Theorem  2.1 

A necessary  and  sufficient  condition  for  the  convergence  of 
the  fixed  point  iteration  given  by: 


Vl  * B V* 


is  that  the  magnitude  of  the  eigenvalues  of  the  iteration  matrix 
B be  less  than  one  [15 J. 

Applying  Theorem  2.1  to  Equation  (2.24)  it  can  be  seen 
that  the  sequence  h^  produced  by  the  LMS  Gradient  Method  will 
converge  in  the  mean  to  h*  if  and  only  if  all  the  eigenvalues 


of  the  matrix: 


B > (1-2,14^) 


are  strictly  less  than  one  in  absolute  value.  Now,  let  X^  be  an 
eigenvalue  of  $ corresponding  to  the  eigenvector  . Then: 


U-^xx^i  = V2|lVi 


(I-2y$xx)4'i  = ¥i-2yXi¥i  (2.26) 


(I-2y$xx)'Fi  = (l-2yXi)'Pi 


From  (2.26)  it  can  be  seen  that  (l-2yX^)  is  an  eigenvalue  of 
(I-2y$xx)  with  corresponding  eigenvector  Thus,  by  Theorem 

2.1,  the  LMS  Gradient  Method  will  converge  in  the  mean  for  any 
value  of  y such  that  the  following  inequality  is  satisfied  for 
all  X.  which  are  eigenvalues  of  $ : 

X AA 


l-2yX.|  . < 1 


(2.27) 


It  was  shown  in  Chapter  I that  the  matrix  was  positive  defin- 
ite. Therefore,  all  of  its  eigenvalues  are  greater  than  zero. 
Thus,  (2.27)  can  be  written  as: 


0 < 2yXi  < 2 


(2.28) 


1 

. 

— 

— 

1 1 
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Since  (2.28)  must  hold  for  all  the  eigenvalues  of  4^,  y must 
satisfy  the  following  inequality  if  the  LMS  Gradient  Method  is 
to  converge  in  the  mean: 

0.  < y . < r-i-  . (2.29) 

max 


where : is  the  largest  eigenvalue 


uj.  une  auLU^iiexa- 


matrix  <5^ . 


Equation  (2.29)  provides  a criterion  for  the  selection  of 
the  parameter  y to  insure  convergence  in  the  mean  of  the  LMS 
Gradient  procedure.  However,  since  no  knowledge  of  the  input 
autocorrelation  matrix  is  assumed,  (2.29)  is  of  little  practical 
value.  In  practice,  y is  usually  chosen  to  be  a "small"  positive 
number  to  insure  that  (2.29)  is  satisfied.  However,  as  will  be 
shown  in  the  next  section,  the  parameter  y directly  effects  the 
rate  of  convergence  of  the  LMS  Gradient  Algorithm  so  that  the 
choice  of  a very  small  value  for  y may  have  an  adverse  effect  on 
the  rate  of  convergence. 


2.3  Rate  of  Convergence  of  the  LMS  Gradient  Algorithm 

It  has  been  shown  that  under  certain  conditions  the 

sequence  h produced  by  the  LMS  Gradient  Algorithm  converges 
— n 

in  the  mean  to  the  optimal  unit-sample  response  h* . That  is , h 
is  an  asymptotically  unbiased  estimator  of  h*  provided  the 
value  of  the  parameter  y is  such  that  (2.29)  is  satisfied. 
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Thus,  it  would  appear  obvious  to  consider  the  rate  at  which  the 
convergence  takes  place;  or,  how  "fast"  does  Eth^  approach  h*. 
However,  since  the  desired  result  of  the  signal  processing 
considered  here  is  to  produce  an  approximation  to  the  minimum 
MSE  signal  estimate,  it  is  of  more  interest  to  consider  the  rate 
at  which  the  MSE  of  the  LMS  Gradient  Algorithm's  signal  estimate 
approaches  the  minimum  MSE  produced  by  the  optimal  filter.  The 
MSE  for  the  LMS  Gradient  Algorithm  is  given  by: 

E [ (S  (n)  -S  (n) ) 2 ] = E [S2  (n)  ]-2E  [S  (n)  S (nl  +ElS2(n)]  . 

Substituting  for  S(n)  from  (1.7)  yields: 

E [(S(n)-S(n)  )2  ]=E  [S2(n)  ]-2E  [S  (n)  XT  (n-n. ) h 1+E  [hTX  (n-n. ) XT  (n-n. ) h ] 

— i — n — n—  1 — 1 — n 

(2.30) 

If  the  covariance  between  X(n-n^)  and  is  neglected  as  well  as 
the  covariance  of  hn(i),  hn(j),  Xfn-n^-i)  and  X(n-n1~j)  then 

(2.30)  can  be  rewritten  as: 

MSE(n)*E[S2  (n)  ]-2E  [S  (n)  XT  (n-^)  ]E  [h^+E  [h^]E  [X(n-n]_)  XT  (n-^)  )E  [h^  ] 

(2.31) 

Justification  for  this  assumption  is  offered  in  Appendix  II. 
Applying  definitions  (1.11)  and  (1.12)  to  (2.311  yields: 

MSE  (n)  = E[s2(n)  ]-2RjYE[h  ]+E[h^]$YYE[h  ] . (2.32) 

— oa  — n — n AA  - n 
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Equation  (2.32)  then'  gives  an  approximation  to  the  mean-square 
error  of  the  LMS  Gradient  procedure. 

The  MSE  produced  by  the  optimal  filter  h*  is  given  by: 

MSE*  (n)  = E[S2 (n)l -2RgXh*+h*T$xxh*  . (2.33) 

Thus,  the  difference  between  the  minimum  mean-square  error 
MSE (n)  and  the  estimate  of  the  mean-square  error  of  the  LMS 
Gradient  procedure  given  by  (2.32)  may  be  written  as: 

MSE  (n)  -MSE*  (n)  = E[h^]  i^Efh^] -2RgXE(hn] -h*T$xxh*-t-2RgXh* 

MSE (n) -MSE* (n)  = (Eth  ] -h*) T$vv (E[h  ] +h*) -2Rgv (E [h  ] -h*) 

— n — ax  — n — — bx  — n — 

or: 

MSE  (n) -MSE*  (n)  = (Eth^l -h*)  T(  »XXE  [1^1 +*^*-2^)  (2.34) 


where  A is  the  diagonal  matrix  containing  the  eigenvalues  of 
4>xx  along  the  diagonal.  Thus,  (2.38)  may  be  expressed  as: 


MSE(n)-MSEMn) 


(h  -h*)TA  (h  -h*) 
— n — — n — 


(2.42) 


Since  A is  a diagonal  matrix,  (2.42)  can  be  rewritten  as: 

MSE (n) -MSE* (n)  = l A . (h_ (i) -h* (i) ; (2.43) 

i-1  1 n 

where  the  X.  are  the  eigenvalues  of  and  h (i)  and  l^(i)  are 
i xa  n 

the  expected  uncoupled  unit-sample  response  of  the  LMS  Gradient 
Algorithm  and  the  uncoupled  unit  sample  response  of  the  optimal 
filter  respectively.  Now,  recall  from  (2.24)  that: 


* Vi  ’ (I-2utxx)Et!!ni+2l,£xx  • (2-24) 

Substituting  the  definition  (2.37)  into  (2.24)  yields  the  fol- 
lowing iteration  equation  for  the  uncoupled  variables : 

Vi  - mT<i-2'j\x)mV2uM\x 

or: 

h^  * (I-2uA  )hn+2uMTRxx  . (2.44) 


The  optimal  unit  sample  response  vector  h*  is  a fixed  point  of 
the  iteration  equation  (2.24)  provided  condition  (2.29)  on  the 
value  of  the  parameter  u is  met.  That  is: 


h*  = (I-2y$xx)h*+2yRxx 
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(2.45) 


Thus,  in  terms  of  the  uncoupled  variables,  (2.45)  can  be  re- 
written as: 


h*  * (I-2yA  )h*+2yM  Hxx  . 


(2.46) 


Therefore,  the  difference  between  the  uncoupled  unit  sample 
response  of  the  LMS  Gradient  procedure  and  the  optimal  filter 
is  given  by: 


(h  -h*)  = (I-2y/U(h  ,h*)  . 

— n — — n-±— 


(2.47) 


The  relationship  in  (2.47)  may  also  be  expressed  in  a non- 
recursive form  as : 


(h^-h*)  = (I-2yA  )n(h0-h*) 


(2.48) 


where  h^  is  the  initial  uncoupled  unit-sample  response  of  the  LMS 
Gradient  Algorithm.  Using  (2.43)  and  (2.48)  the  difference 
between  the  approximate  mean  square  error  of  the  LMS  Gradient 
procedure  and  the  minimum  MSE  can  be  expressed  as : 

N , 

MSE  (n) -MSE*  (n)  = Z X . (l-2uX, ) (hn (i) -h* (i) ) . (2.49) 

i=l  1 1 u 

The  relationship  given  in  (2.49)  is  the  fundamental  re- 


sult of  this  section.  It  places  a bound  on  the  rate  of  converg- 
ence of  the  LMS  Gradient  procedure.  Note  that  this  relation 


again  shows  that  for  convergence  of  the  algorithm  (i.e.  for 
convergence  of  MSE(n)  to  MSE*(n)): 

(l-2uXi)  < 1 V-  xi 

This  is  consistent  with  the  results  derived  earlier.  Equation 
(2.49)  gives  a lower  bound  on  the  rate  on  convergence  of  the  LMS 
Gradient  Procedure  in  that  convergence  will  proceed  according  to 
(2.49)  only  if  precise  knowledge  of  the  gradient  is  used  in 
the  algorithm  rather  than  the  estimate.  While  the  estimate  of 
tne  gradient  used  in  the  LMS  Gradient  Algorithm  is  unbiased  as 
discussed  in  Section  2.1.1,  it  does  have  finite  variance.  Thus, 
the  LMS  Gradient  Algorithm  will  always  produce  a mean-square 
error  which  converges  to  the  minimum  at  a rate  which  is  slower 
than  that  indicated  by  (2.49).  Also,  it  is  emphasized  that  the 
results  of  this  section  are  valid  only  for  cases  in  which  the 
autocorrelation  matrix  and  the  autocorrelation  vector  are 
independent  of  time.  While  the  bound  on  the  rate  of  convergence 
given  by  (2.49)  is  approximate  and  subject  to  several  constraints, 
it  is  still  useful  for  the  comparison  of  the  LMS  Gradient  Algor- 
ithm to  the  Adaptive  Fixed-Point  Iteration  Algorithms  discussed 
in  the  next  Chapter.  The  relationship  between  the  theoretical 
and  actual  convergence  properties  of  the  algorithm  will  be  in- 
vestigated via  computer  simulations  in  Chapter  VI. 
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2 . 4 LMS  Gradient  Algorithm  — Another  View 

The  LMS  Gradient  Algorithm  developed  by  Widrow  and  dis- 
cussed in  the  previous  sections  is  based  on  the  use  of  the 
Method  of  Steepest  Descent  for  the  minimization  of  the  mean- 
square  error  function.  Since  the  gradient  of  the  mean-square 


error  is  unknown,  an  estimate  of  the  gradient  is  used  in  the 
algorithm.  The  convergence  of  the  LMS  Gradient  procedure's  unit 
sample  response  to  the  optimal  implies  that  the  algorithm  has 
somehow  adapted  to,  or  "learned"  something  about,  the  statistics 
of  the  data.  However,  it  is  difficult  to  see  from  the  relation- 
ships which  make  up  the  algorithm  how  the  estimates  of  the 
statistics  are  obtained.  In  this  section,  an  algorithm  will 
be  proposed  based  on  the  orthogonality  principle  and  direct 
estimates  of  the  statistics.  It  will  be  shown  that  this  algor- 
ithm is  almost  identical  to  the  LMS  Gradient  Algorithm. 

The  orthogonality  principle  states  that  the  signal  esti- 
mate must  be  such  that  the  estimation  error  is  orthogonal  to 
the  data  for  minimum  mean-square  error.  That  is,  the  vector 
identity: 

E[X(n-ni)  (S(n)-S(n) ) ] = 0 (1.9) 

must  be  satisfied.  Expanding  this  identity  gives: 

E [X  (n-n1)  S (n)  ] - E [X  (n-n^  S (n)  ] = 0 
and,  recognizing  that  S (n) =X (n) -N (n)  yields: 


5 

E(X(n-n1)X(n)]-E[X(n-n1)N(n)]-E[X(n-n1)S(n)]  = 0.  (2.50) 

Now,  applying  the  assumptions  used  in  Chapter  I that  the  signal 
and  the  noise  are  orthogonal  and  that  noise  samples  separated 
by  n1  or  more  samples  are  also  orthogonal  allows  (2.50)  to  be 
rewritten  as : 

E(X (n-n^) X (n) ] -E(X (n-n^) S (n ) 1 =0  . (2.51) 

If  a priori  knowledge  of  the  statistics  of  the  data  was  available 
(2.51)  could  be  solved  for  the  unit  sample  response.  Now,  sup- 
pose that  direct  estimates  are  made  of  the  two  expected  value 
terms  in  (2.51)  and  that  these  estimates  are  based  on  the  past 
data  through  X(n-l)  and  the  past  signal  estimates  through  S (n) . 
That  is  define  two  estimates  Y^  and  Y2  as  follows: 

/\  1 n— 1 

Y = — E X(£-n. )X(£)  (2.52) 

-1  n 2=0  ~ 1 


Y2  = jjiy  E XU-n^SU)  • (2.53) 

If  the  statistics  of  the  data  are  stationary  and  if  S (n)  is  the 

/V  A 

minimum  mean-square  error  estimate  then  Y^  and  Y2  are  consistent 
estimates  of  the  two  expected  value  terms  in  (2.51).  Thus,  as 
n becomes  large,  the  estimates  (2.52)  and  (2.53)  should  come 
close  to  satisfying  the  equality  of  (2.51).  That  is: 
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n 


1 n_  ^ 1 

r-  E X|H,)X(i)  - -XT  E X(4-n  )S(4) 
h Jl=0  n+1  £_q  1 


« 0 


or: 


Xln-n.  )S  (n)  - E x(4-n. ) (*—■  X(4)-S(4) ) 
1 i=o  ~ 1 n 


* 0.  (2.54) 


exactly  satisfy  (2.54).  However,  at  least  formally,  each  side 


of  (2.54)  may  be  pre-multiplied  by  X (n-i^)  and  divided  by  the 

.T 


scalar  X (n-n^)  X (n-n^)  . After  this  manipulation,  (2.54)  may  be 
written  as : 


S (n)  = 


-= XT  (n-n. ) E X(i-n.)  (S±i  X(4)-S(4>). 

X (n-n.)X(n-n, ) 1 4=0  L 


1 — 1‘ 


(2.55) 


Now,  suppose  that  n is  large  enough  so  that: 


n+1  „ 


n 


Then,  (2.55)  can  be  rewritten  as: 


n-1 


S (n)  = 


X (n-n^)  X (n-n^ 


Equation  (2.56)  gives  a relationship  for  a signal  estimate 
based  on  an  attempt  to  satisfy  the  orthogonality  principle  but 
substituting  the  direct  estimates  (2.52)  and  (2.53)  for  the 
expected  value  terms.  However,  (2.56)  would  be  more  attractive 


With  the  past  data  and  signal  estimate  samples  available,  (2.54) 
may  be  considered  as  a set  of  N linear  equations  in  one  unknown 

A /\ 

S(n).  Thus,  there  may  or  may  not  be  an  estimate  S(n)  which  will 


X (n-n. ) E X (4-n. ) (X  (4) -S (4) ) . (2,56) 
1 4=0 
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if  if  were  in  a recursive  form.  The  relationship  (2.56)  may  be 
placed  in  a recursive  form  by  defining  the  N-vector  h as: 


n-1 


X (n-n^) X (n-n^)  1=0 


E Xd-x^)  (X(A)-SU)).  (2.57) 


Thus,  (2.56)  can  be  rewritten  as: 


n. 


S(n)  = X (n-n,)h  = E h(i)X(n-i). 

- 1 - i-n, 


(2.58) 


Equation  (2.58)  is  in  the  form  of  a finite  impulse  response 
digital  filter  as  is  used  for  estimation  in  the  LMS  Gradient 
procedure.  The  unit-sample  response  can  be  updated  recursively 
by  considering  equation  (2.57)  and  noting  that:  • 


n 


^n+1 


X (n+l-n1)X(n+l-n1)  1=0 


E X()l-n1)  (XU)-S(4) ) 


or: 


X (n-n^  Xfn-n^) 


-h  +- 


— n + 1 T tin  ' T 

X (n+l-n^Xtn+l-n^^1  X (n+l-n^Xfn+l-n^ 


(X(n)-S(n)  )X(n-nL) 


(2.59) 


Note  from  (2.50)  that  the  equation  for  updating  the  unit  sample 
response  in  this  algorithm  is  very  similar  to  that  used  in  the 
LMS  Gradient  Method  (2.18)  except  that  data  dependent  coeffi- 
cients are  used  in  the  recursion  relation.  Now,  assume  that  X(n) 
is  a stationary  process  and  consider  the  term: 


I 
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X1  (n-n.JXfn-n.)  = l X (n-i) . 

i=nl 

The  terms  XT  (n-n^  X (n-n^  and  XT  (n+1-n.^X  (n+l-n.^)  are  the 
squares  of  the  norms  of  two  N-vectors  which  differ  in  magnitude 
due  to  only  one  component.  That  is: 


XT(n-n1)X(n-n1)  = XT  (n+l-n^  X (n+l-n.^  +X2  (n+1-^ ) -X2  (n+l-n^ 
or: 

XT (n-n, ) X (n-n, ) X2 (n+l-N-n, ) -X2 (n+l-n, ) 

_= l-Z i = 1 + = i ±-  . (2.61) 

X (n+lrn1)X(n+l-n1)  X (n+l-n.^  X (n+l-n^ 

Now,  if  the  probability  density  function  of  X(n-n,)  is  rela- 
tively  concentrated  near  the  center  of  gravity  E (X(n-n1) ] of 
the  distribution,  then  it  can  be  shown  that  to  a second  order 
approximation : 

XT (n-n, ) X (n-n, ) 

E [-= i ] = 1 (2.62) 

X (n+l-n^X  (n+l-nj^) 

and  that  the  variance  of  this  quotient  is  small  if  N is  rela- 
tively large.  Appendix  C gives  a proof  of  these  statements. 
Since  the  mean  of  the  term  multiplying  h^  in  equation  (2.59) 
is  one  and  its  variance  is  small  for  relatively  large  values 
of  N,  substituting  unity  for  this  term  yields  a good  approxima- 


tion. Thus,  (2.59)  can  be  written: 


5 


^n+1 


+ 


(n+l-n^) X (n+l-n^) 


(X(n)-S(n)  )X(n-n1)  . 


(2.63) 


It  can  be  seen  from  (2.63)  that  under  the  approximation 

discussed  above  the  updating  of  the  unit-sample  response  under 

this  algorithm  is  identical  to  that  under  the  LMS  Gradient 

Algorithm  except  that  the  constant  2y  is  replaced  by  the  data 

T -1 

dependent  term  (x  (n+l-n^Xfa+l-n^) ) . New,  it  has  been  shown  previous- 

ly that  the  LMS  Gradient  Algorithm  will  converge  for  any  value 

of  y such  that  (2.29)  is  satisfied.  That  is,  if  X is  the 

max 

largest  eigenvalue  of  the  input  autocorrelation  matrix,  then  for 
convergence  the  following  inequality  must  hold: 


0 


< y 


< 


_l 

Vax 


The  trace  of  a matrix  is  defined  as  the  sum  of  its  eigenvalues 
and  may  also  be  computed  as  the  sum  of  its  diagonal  elements. 
Thus,  since  all  the  eigenvalues  of  $xx  are  greater  than  zero 
as  shown  in  Chapter  I : 


max 


N 

< I 
i=l 


X. 

i 


N-l  2 

£ E [X  (n-n.-i)  ] 
i=0  x 


Thus,  the  LMS  Gradient  method  will  converge  if  y is  chosen  so 
that: 

1 N_1  2 
± > £ E [X  (n-n.-i)  ] 

y i=l  A 

Or,  since  the  process  is  assumed  stationary: 


^ > N E X2(n) 
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Now,  consider  the  term  X1  (n+l-n^)  X (n+l-n^  as  given  by  (2.60) 
and  take  the  expected  value  of  both  sides: 


“2"1 


E [XT(n+l-n1)X(n+l-n1)  ] = I E tX2  (n-i)  1 = N ElX2(n)l  . 


i=n^-l 


Thus,  the  term  X (n+l-n^  X (n+l-n^  is  an  unbiased  estimate  of 
N *E  [x  (n)  ].  Therefore  the  recursion  relation  (2.63)  for  the 
unit  sample  response  is  identical  to  that  for  the  LMS  Gradient 
procedure  except  that  the  constant  2 y has  been  replaced  by  an 
estimate : 


2y  = 


X (n+l-n^Xto+l-n.^ 


of  a step  size  along  the  gradient  which  would  guarantee  con- 
vergence if  its  true  value  were  known. 

Thus,  with  the  assumptions  mentioned  above,  the  algorithm 
presented  here  based  on  an  attempt  to  make  direct  estimates 
(2.52)  and  (2.53)  approximately  satisfy  the  orthogonality 
principle  may  be  stated  as  follows : 


S (n)  = X (n-n. )h 
— 1 — n 


h_^,  = h_+2y(X(n)-S(n))X  (n-n, ) 


X (n+l-n, ) X (n+l-n.  ) 


(2.64) 


(2.65) 


2.66 


T 
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It  is  obvious  from  the  above  equations  that  the  algorithm 
suggested  here  is  very  similar  to  the  LMS  Gradient  Method  and, 
in  fact,  may  be  considered  as  a modification  of  the  LMS  Gradient 
Method  to  include  an  adaptive  step  size.  Thus,  an  algorithm 
almost  identical  to  the  LMS  Gradient  Method  has  been  obtained 
by  direct  estimation  of  expected  values  and  application  of  the 
orthogonality  principle  of  the  estimates. 

2 . 5 Conclusion 

The  LMS  Gradient  Algorithm  as  developed  by  Widrow  is  an 
adaptive  recursive  algorithm  for  the  approximate  solution  of  the 
discrete  Weiner*-Hopf  equation  for  a finite  impulse  response 
digital  filter.  The  LMS  Gradient  Method  is  based  on  the  Method 
of  Steepest  Descent  for  minimization  of  a function  and  can  be 
shown  to  converge  in  the  mean  provided  that  the  step  size  along 
the  gradient  lies  within  a certain  bound.  A bound  on  the  rate 
of  convergence  of  the  LMS  Gradient  has  been  derived.  This 
bound  relates  the  eigenvalues  of  the  input  autocorrelation 
matrix  4>xx  to  the  rate  at  which  the  mean-square  error  of  estima- 
tion produced  by  the  LMS  Gradient  Algorithm  approaches  the  mini- 
mum mean-square  error  produced  by  the  optimal  filter.  The  LMS 
Gradient  Algorithm  will  cpnverge  at  a rate  less  than  or  equal 
to  this  bound. 

The  LMS  Gradient  Method  was  developed  based  on  the  Method 


of  Steepest  Descent  for  the  minimization  of  the  mean-square 
error  function.  An  estimate  of  the  gradient  was  used  in 
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developing  the  Steepest  Descent  recursion  relation.  In  this 
chapter,  it  was  proposed  that  an  algorithm  for  estimation  be 
based  on  direct  estimates  of  expected  values.  It  was  shown  that, 
given  certain  simplifying  assumptions,  the  resulting  algorithm 
closely  resembled  the  LMS  Gradient  Method  except  that  a data- 
dependent  step  size  was  used  rather  than  the  fixed  parameter  u 
of  the  LMS  Gradient  Algorithm.  However,  this  algorithm  was 
based  only  on  direct  estimates  of  the  statistics  and  application 
of  the  orthogonality  principle.  No  numerical  analysis  methods, 
such  as  Steepest  Descent  were  used.  This  development  offers 
another  view  of  the  LMS  Gradient  Algorithm. 


CHAPTER  III 

ADAPTIVE  ESTIMATION  USING  FIXED  POINT  ITERATION 

The  LMS  Gradient  Algorithm  offers  an  alternative  to  the 
direct  inversion  of  am  estimated  autocorrelation  matrix  in  the 
approximate  solution  of  the  discrete  finite  impulse  response 
Wiener-Hopf  equation  (1.13).  However,  as  has  been  previously 
mentioned,  the  choice  of  the  parameter  y has  a significant 


effect  on  the  convergence  properties  of  the  algorithm.  The 
choice  of  a "small"  value  for  y to  insure  convergence  has  a 
determental  effect  on  the  rate  of  convergence  as  was  shown  in 
Chapter  II.  In  this  chapter,  other  algorithms  based  on  fixed- 
point  iteration  methods  are  studied,  j These  methods  prove  to  be 
superior  to  the  LMS  Gradient  Method  in  terms  of  rate  of  con- 
vergence. The  fixed-point  iteration  (algorithms  considered  here 

may  be  divided  into  two  parts.  One  part  is  the  estimation  of  the 

/ 

I 

autocorrelation  matrix  and  autocorrelation  vector  or  the  "adapt- 

i 

ation"  to  the  statistics  of  the  inpiit  process.  The  second 

* 

part  is  the  recursive  algorithm  which  uses  the  estimates  of  4>xx 
and  R^  to  determine  the  unit-sample  response  h(n).  In  this 
chapter,  both  the  estimation  of  $^x  and  and  the  approximate 
solution  of  the  resulting  linear  System  by  fixed  point  iteration 

t 

are  considered.  The  consistency  of  the  chosen  estimates  of  $YY 
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an<3  is  considered  in  detail  for  the  case  of  a stationary 
input  process  X(n)  and  the  case  of  time  varying  statistics  is 
also  discussed.  The  convergence  properties  of  fixed-point 
iteration  algorithms  is  explored  in  the  general  case  in  this 
chapter  and  two  particular  algorithms  are  considered  in  detail. 
These  two  algorithms  are  based  on  the  familiar  Jacobi  Relaxation 
and  Successive  Over- Relaxation  Methods  respectively.  The  methods 
based  on  these  fixed-point  iteration  algorithms  offer  higher 
rates  of  convergence  them  the  IMS  Gradient  Algorithm  at  the 
expense  of  the  number  of  operations  required  to  implement  the 
procedure.  Methods  for  reducing  the  number  of  operations  re- 
quired by  Jacobi  Relaxation  and  Successive  Over- Relaxation 
are  considered  in  the  final  section  of  this  chapter.  • 


3.1  Estimation  of  the  Autocorrelation  Matrix  and  Vector 

As  mentioned  previously,  one  task  of  an  adaptive  processor 
is  to  make  some  estimate  of  the  statistics  of  the  input  process. 
In  the  case  of  the  LMS  Gradient  Algorithm,  the  gradient  of  the 
mean  square  error  was  estimated  by  (2.17)  which  proved  to  be  an 
unbiased  estimate.  In  the  fixed-point  iteration  algorithms 
considered  in  this  chapter,  the  problem  of  adaptation  is  ap- 
proached by  the  use  of  direct  estimates  of  the  input  auto- 
correlation matrix  and  the  input  autocorrelation  vector.  The 
estimates  used  are  shown  to  be  consistent  for  processes  for 
which  a Gaussian  approximation  is  reasonable.  In  addition,  a 
bound  on  the  variance  of  these  estimates  is  given.  Throughout 
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the  following  discussion/  the  input  process  X(n)  is  assumed  to 
be  stationary.  The  non-st at ionary  case  is  discussed  at  the  end 
of  this  section. 


The  estimates  of  the  autocorrelation  matrix  and  the  auto- 
correlation vector  of  the  input  process  X(n)  considered  here  are 
the  direct  estimates  given  by: 


<j>.  . 

J-3 


i M-1 

= £ X (n-i)  X (n- j ) 

M n=0 


(3.1) 


R. 

i 


, M-l 

— E X(n) X(n-n. -i+1) 
M n=0  1 


(3.2) 


th 


where  <Jk  j and  R^  are  the  (i,j)  element  of  the  autocorrelation 


. th 


matrix  estimate  and  the  i component  of  the  autocorrelation 


vector  estimate  respectively.  Equations  (3.1)  and  (3.2)  are 


4-  u 

estimates  of  the  (i,j)  element  of  4>xx  and  the  i component 


of  R xx  respectively  whose  true  values  are  given  by: 


* E (X(n-i)X(n-j)  ] 


(3.3) 


Ri  » E [X (n) X (n-n^-i+1) ] 


(3.4) 


Since  the  input  process  has  been  assumed  to  be  stationary,  the 


autocorrelation  matrix  estimate  $xx  and  the  autocorrelation 


vector  estimate  Rxx  are  made  up  of  samples  of  the  sample  auto- 
correlation function  is  given  by: 


i 
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*=C(k,M 


1 

M 


M-l 

Z 

n=0 


X (n)  X (n- 1 k | ) 


(3.5) 


where  it  is  assumed  that  data  is  available  for  all 
-(n^+N-1)  n < m-l.  Note  that  the  symmetry  of  the  sample  auto- 
correlation function  has  been  indicated  by  the  use  of  |k|  on  the 
right-hand  side.  Since  the  estimates  (3.1)  and  (3.2)  cam  be 
obtained  from  (3.5)  as: 


*ij  RXX*x~^M  (3'6) 

\ ~ RXX(nl+i_1)M  (3.7) 


the  properties  of  the  estimate  given  by  (3.5)  will  be  considered 
here.  Specifically,  the  bias  and  variance  of  (3.5)  is  of 
interest.  If  the  estimate  (3.5)  is  consistent  (i.e.,  if  it  is 
unbiased  and  its  variance  approaches  zero  for  large  M)  then 
Rxx(|k|)  converges  to  the  true  autocorrelation  function  with 
probability  one.  That  is  for  all  e > 0: 

lim  p(|Rxx(k)M-Rxx(k)  | <e)  - 1 

M-*» 

Taking  the  expected  value  of  both  sides  of  (3.5)  yields: 


E [Exx(1c)m] 


M-l 

Z 

n=0 


X(n)X(n-|k| ) 1 


(3.8) 


By  the  linearity  of  the  expected  value  operator,  (3.8)  can  be 
written  as : 
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! 


Et‘Wk>M1 


, M-l 
i r 
M n 
n=0 


E [X(n)  X (n-  | k | ) ] 


or,  since  X(n)  is  stationary: 


E[^XX(k)M]  = ElX(n)X(n-|k|)  ] = R^fk)  . (3.9) 

As  can  be  seen  from  (3.9)  EfR^fk)^  is  equal  to  the  true  value 
of  the  autocorrelation  function  (k ) . Thus,  by  definition, 
the  estimate  given  by  (3.5)  is  unbiased. 

The  bias  of  R^tk^  was  determined  above  with  relatively 
little  effort.  However,  toe  variance  of  R^tk^  dd^ficult  to 
determine  in  the  general  case  unless  the  fourth-order  statistics 
of  the  process  X(n)  are  known.  Thus,  it  will  be  assumed  that 
the  sample  values  of  the  process  X(n)  are  jointly  Gaussian. 
However,  it  can  be  shown  that  in  most  cases  the  use  of  the 
Gaussian  assumption  yields  a good  approximation  [16] , [17]. 

The  variance  of  Rxx(k)  may  be  written  as  follows: 

VARfR^fk)^  » E(  (Rxx(k)M-E[Rxx(k)M]  )2]  (3.10) 

or: 

VAR(gXX(k)Ml  ” Ef5it(k)Ml-2E[kXX(k)Ml2+E[SXX<k)MJ2  ' 

Thus,  combining  terms  in  the  above  equation  and  substituting 
from  (3.9)  yields: 
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VAR[Rxx(k)M]  = El*xx(k)M1_RXX(k)  * (3.11) 

Now,  substituting  the  definition  of  Rxx^m  from  (3.5)  into 
(3.11)  the  following  expression  is  obtained: 

„ i M-l  , M-l  - 

VAR[Rxx(k)Ml=E[  (i  E X(j)X(j-|k|)) (q  Z X (i)  X (i- 1 k | ) ) ] -R^  (k) 

or: 

* . M-l  M-l  2 

VAR[Rxx(k)M]  = E[±j  ^ ^ X(j)X(j-|k|)X(i)X(i-|k|)]-Rjx(k). 

Applying  the  linearity  of  the  expected  value  operator  yields : 

„ , M-l  M-l  2 

VAR(Rxx(k)M)  = E E E[X(j)X(j-|k|  )X(i)X(i-|k|  )]-R^(k)  . 

(3.12) 


Note  from  (3.12)  that  the  variance  of  the  autocorrelation 
estimate  depends  on  the  fourth  order  statistical  expression 
E[X(j)X(j— |k| )X(i)X( i— | k | ) ] - At  this  point,  it  will  be  assumed 
that  the  processes  X(i)  and  X ( j ) are  jointly  Gaussian.  In  this 
case  the  fourth-order  statistical  expression  in  (3.12)  is  given 
by: 


E [X  ( j ) X ( j- | k | ) X (i)  X (i- | k | ) ] =R^X  (k) +R^ (i- j ) +RXX  (i- j- | k | ) 
R^  (i- j + |k  | ) . 


Thus,  by  substituting  (3.13)  into  (3.12),  the  variance  of 
Rxx(k)  may  be  written: 


(3.13) 


6 


VAR  tRxx(k)M] 


M-l  M-l 


l Z (R^j^U-jJ+Rjjjjd-j-lkl  JR^d-j  + lkl  ) ) 


M j=0  k=0 


(3.14) 


The  Gaussian  assumption  is  not  extremely  restrictive  in  that  it 
can  be  shown  that  (3.14)  gives  a good  approximation  to  the  true 
variance  of  the  estimate  provided  that  certain  constraints  are 
met.  It  can  also  be  shown  that  most  reasonable  processes  meet 
the  required  constraints  [17]. 

Equation  (3.14)  may  be  simplified  further  by  introducing 
a new  variable  n * (i-j)  which  gives: 


VAR 


„ , M-l  M-l- j - 

[RXX(k)M]  = T-  .z  E.  (RxX(n)+RxX(n-|k|  )Rxx(n+|k|  ) ) 
M i=0  n=- j 


or: 


VAR 


M-l  M-l 


Now 


[*XX(k)M]  = T’  .E  Z (RXX(n)+RXX(n-lkl  )RXx(n+|k|)) 
M j*0  n=l-M 


“(j+1)  2 

Z (^(nJ+R^tn-lkl  JR^tn+lkl  ) ) 
n*l-M 


m-l  j 

Z (RxX(n)+Rxx(n-|k|  JR^tn+lIkl  )) ) . (3.15) 

n*  j+1 


, since  R^tn)  =Rxx(-n)  , (3.15)  can  be  rewritten  as: 


65 


* , M-l  M-l  2 

VAR[Rxx(k)M]  =*  ^ [ I (R^x(n)+Rxx(n-|k|)Rxx(n+|k|) 


-2  E (Rjx(n)+Rxx(n-|k|)Rxx(n+|k|))] 
n— j+1 


Interchanging  the  order  of  summation  and  summing  over  j gives: 


VAR[Rxx(k)M]  M ^ (1-  -L-L)  (Rxx(n)+Rxx(n-|k|  )Rxx(n+|k|  ) ) . 


n=l-M 


(3.16) 


Equation  (3.16)  is  an  expression  for  the  variance  of  R^fk^ 
in  terms  of  the  autocorrelation  function  Rxx(k)  of  the  input 
process  X(n).  As  was  stated  earlier,  (3.16)  is  exact  for  zero 
mean  Gaussian  processes  and  approximate  for  most  other  pro- 
cesses of  interest. 

If  rxx(]c)m  is  a consistent  estimate  of  R^fk),  then  its 
variance  given  by  (3.16)  must  approach  zero  as  M becomes  large. 
That  is: 


lim  VAR[Rxx(k)M]  - 0 
M-*<» 


(3.17) 


To  see  that  (3.17)  does  hold,  consider  the  following  limit: 


lim  M VAR[Rxx(k)M)  = E (R^x  (n)  +RXX  (n+ 1 k | ) R^  (n-  | k | ) ) . (3.18) 

M-*°°  n*-« 

2 i 

Now,  if  the  sequences  Rxx  and  RJfX  (n- j k | ) Rxx  (n+ 1 k | ) are  ab- 
solutely summable  then: 


I 


lim  M VAR[R„v(k)M] 
M-mo  XX  M 


< CO 


I 


Thus,  since  the  limit  in  (3.18)  approaches  a finite  constant 
as  M approaches  the  variance  of  the  estimate  must  approach 
zero  as  M becomes  large. 

It  has  been  shown  that  the  estimate  of  the  autocorrela- 
tion function  given  by  (3.5)  is  unbiased  and  its  variance 
approaches  zero  as  M approaches  08 . Thus,  R^fk^  a consistent 
estimate  of  R^Oc). 

A 

The  rate  at  which  the  variance  of  Rxx(k)M  approaches 
zero  earn  be  determined  by  considering  the  expression  for  the 
variance  given  by  (3.16).  From  (3.16): 


VAR[Rxx(k)M+1] 


1 

M+l 


1 M(1"  M+l^RXX^n^+RXX^n“lk^RXX^n+l1Cl  ^ 

n=-M 


/\ 

or,  in  terms  of  VAR[Rxx(k)M] : 

- mSt  VARl6XX<k>M1  + HTT*1”  Ipr'  (Rxx(M)+!Stx 

(M- I k I )Rxx(M+|k| ) ) . (3.19) 


Subtracting  both  sides  of  (3.19)  from  VAR[Rxx(k)M]  yields: 
VAR[Rxx(k)Ml_VARf^xx(k)M+l^  (1  jf^r)  (vAR[Rxx  (k^M3 


' Mil  <*L<M)+Rxx(M|k|)Exx(M+|k|)l 
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or: 


VAR^RXX^M^"VAR^RXX^k)M+]J  M+l  tVAR 


- jjlj-  (rJx(M)  + Rxx  (M- 1 k I ) IM+ 1 1c I ) ) ] 


(3.20) 


From  (3.20)  it  can  be  seen  that  if  (Rxx (M) +RXX (M- | k | ) Rxx (M+ | k | ) ) 
is  small  for  large  values  of  M (which  is  a necessary  condition 
for  the  sequence  to  be  absolutely  summable) , then  the  variance 

A 

of  RxX(k)M  decreases  as  1/M  for  large  values  of  M. 

A 

A bound  of  the  variance  of  R^tk^  can  ^eterm^ne<^  bY 
considering  equation  (3.19).  From  (3.19): 


VAR[Rxx(k)M+i]  = M+1  VAR[^xx(k)M]  + 2 


(M+l)' 


(R^X(M)  + Rxx (M- | k | )Rxx(M+|k| ) ) 


Now,  since  the  autocorrelation  has  a maximum  at  k=0 , the  follow- 
ing inequality  holds  for  all  k: 


VAR[RXX(k)M+l]  - M+l  VAR[RXX(k)M1  + (M+i)2  ^X^  * (3’21) 


For  M=l,  the  variance  of  R^(k)^  is  found  from  (3.16)  to  be: 


VAR[Rxx(k)1]  = R^x  (0)  + R^x  (k)  .<_  2R^X(0) 


(3.22) 
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A 

Thus,  using  (3.22)  and  (3.21)  a bound  for  VAR[Rxx(k)M]  can  be 
generated  sequentially.  This  bound  can  be  used  to  determine  the 

A 

value  of  M which  will  insure  that  the  variance  of  Rxx(k)M  is 
less  than  a specified  level.  Determination  of  an  appropriate 
value  of  M is  discussed  in  the  next  section. 

3.1.1  Estimation  of  the  Autocorrelation  Function  — Time- 
Varying  Case 


L,(k)M  * e xmxu-lki ) • 

XX  M M 4-n-M+l 


(3.23) 


Obviously,  if  R^fk^  to  be  a 9ood  approximation  to  the  time- 
varying  statistic  Rxx(k) , then  (M+n^+N)  should  be  small  as 
possible  while  still  meeting  any  specified  requirements  regard- 
ing the  variance  of  the  estimate.  Ideally,  the  variance  of 


In  the  preceeding  section,  it  was  shown  that,  for  the 

A 

case  of  a stationary  input  process  X(n),  R^fk^  given  by 
(3.5)  is  a consistent  estimate  of  the  autocorrelation  function. 
In  the  case  of  a process  with  time-varying  statistics,  estima- 
tion of  those  statistics  from  the  data  is  difficult.  However, 
if  the  variation  with  time 'of  the  statistics  is  slow  compared 
to  the  number  of  samples  used  in  making  the  estimate  and  to 
the  length  of  the  filter,  then  one  possible  approach  is  to  use 
a sliding  window  of  length  (M+n^+N)  and  compute  a moving 

A 

average.  That  is  at  time  n > M,  the  estimate  R^fk^  wou^d  be 
given  by: 
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RxX(JOm  should  be  very  small  while  keeping  the  value  of 


(M+n^+N)  small  with  respect  to  the  rate  at  which  the  statistic 


R^fk)  is  varying.  Since  the  variance  cannot  be  reduced  without 
increasing  M,  some  engineering  decision  must  be  made.  The 

A 

bound  on  the  variance  of  Bxx^m  which  can  be  computed  using 
(3.21)  and  (3.22)  can  be  used  as  an  aid  in  this  decision. 
Chebychev's  inequality  states  that: 


P[|Rxx(k)M"RxX(k)  I i e]  - 


VAR[RJjx(k)M] 


Thus,  if  B^(M)  denotes  the  bound  on  the  variance  of  Bxx^M 
computed  using  (3.21)  and  (3.22)  then  the  following  inequality 
holds : 


B,  (M) 


P ^ ' RXX^M_RXX  ^k^  — - 2 


or: 


. Bk(M) 

P ( | Rxx(k)M_Rxx(k^  I - i 1 “ 2 


(3.24) 


Equation  (3.24)  can  be  used  to  determine  a bound  on  the  prob- 

/s 

ability  that  the  random  variable  Bxx^m  ^es  a specified 

e-region  of  the  true  value  R^fk)  or  to  determine  the  value  of 
M required  to  insure  that  P [ 1 (k) M~ Rxx (k)  | _<  e]  meets  or 

exceeds  a certain  value  for  a specified  value  of  e. 

Thus,  the  constraints  on  the  selection  of  an  appropriate 
value  of  M are  that  the  variation  of  the  statistic  R^fk)  be 
"small"  over  (M+n-^+N)  units  of  time  and  that  the  value  of  the 


right-hand  side  of  (3.24)  be  close  to  1 for  some  "small"  value 
of  e.  The  definition  of  "small"  is  an  engineering  decision 
and  will  vary  over  a range  of  problems.  Obviously,  in  all  cases 
the  constraints  on  the  value  of  M cannot  be  met.  Thus,  trade- 
offs must  be  made  in  some  situations. 

It  should  be  noted  that  the  selection  of  a value  of  M 
as  described  in  this  section  requires  more  knowledge  about  the 
statistics  of  the  data  process  X(n)  than  was  originally  assumed 
to  be  given.  Specifically,  some  information  about  the  rate  at 
which  the  statistics  vary  with  time  and  some  knowledge  of 
R^tO)  is  required  to  select  M.  However,  the  restrictions 
imposed  by  requiring  this  a priori  knowledge  are  not  great 
in  a large  number  of  practical  problems. 

It- has  been  proposed  that  a moving  average  given  by 
(3.23)  be  used  to  estimate  the  statistic  Rxx(k)  in  the  non- 
stationary case  under  the  constraint  that  R^fk)  be  approximate- 
ly stationary  over  (M+n-^+N)  units  of  time.  If  the  assumption 
of  local  time  invariance  is  valid,  then  (3.23)  is  a consistent 
estimate  of  the  locally  stationary  statistic  Rxx(k)  with 
variance  given  by  (3.16).  The  bound  which  can  be  computed 
using  (3.21)  and  (3.22)  can  be  used  along  with  (3.24)  to  aid  in 
the  selection  of  the  length  M of  the  sliding  window  for  the 
moving  average  provided  that  some  a priori  knowledge  of  the 

(k)  and  R^CO)  is  available. 


time  variation  of  Rxx 


3.1.2  Recursive  Estimation 


The  moving  average  given  by  (3.23)  was  proposed  in  the 
preceeding  section  for  the  estimation  of  the  autocorrelation 
of  a non-stationary  process  whose  statistics  vary  slowly  with 
respect  to  the  length  of  the  data  window  used  in  the  moving 
average.  In  this  section,  a method  for  computing  the  estimate 
given  by  (3.23)  will  be  presented  which  is  computationally 

A 

more  efficient  than  direct  evaluation  of  the  Siam.  If  Rvv  (k) 

xx  ,n 

is  defined  as  the  estimate  of  the  autocorrelation  function  at 
sample  time  n as  given  by  (3.23),  then  at  sample  time  n+1  the 
estimate  is  given  by: 


RXX,n+l(k)M 


l’ 

M 


n+1 

E 

Jl=n-M+2 


X:f£)X(£-|k| ) 


or: 


RyY  n+l(k)M  = 1 X(£)X(£- | k | ) + rj  X (n+1) X (n+1- | k | ) 

XX, n+1  M M A=n_M+1  M 

- i X(n-m+l)X(n-M+l-|k| ) . 


A 

Thus,  R^x  n<k)M  can  be  written  in  recursive  form  as: 


*XX,n+l(k)M  *XX,n(k)M  + M [x<n+1>x(n+1-ikl ) 


- X (n-M+1) X (n-M+1- | k | ) ] 


(3.25 
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The  recursive  form  (3.25)  may  be  used  to  compute  and 

provides  substantial  savings  in  computation  time  over  the  direct 
evaluation  of  the  sum  in  (3.23).  Thus,  the  form  of  (3.25)  will 
be  used  in  the  Adaptive  Fixed-Point  Iteration  Algorithms. 

3.2  Approximate  Solution  of  the  Wiener-Hopf  Equation  By 
Adaptive  Fixed-Point  Iteration 

Historically,  there  are  two  widely  used  methods  for  the 
approximate  solution  of  the  discrete  FIR  form  of  the  Wiener- 
Hopf  equation.  One  method  is  an  adaptive  algorithm  such  as  the 
LMS  Gradient  procedure  discussed  in  Chapter  u.  Another  approach 
is  to  receive  a block  of  input  data,  make  some  estimate  of  the 
autocorrelation  matrix  and  autocorrelation  vector  based  on  this 
block  of  data  and  then  to  solve  the  system  of  equations : 

*XX  - = Bxx  • (3.26) 

The  solution  of  (3.26)  is  then  used  to  filter  the  block  of  data 
to  obtain  an  estimate  of  the  signal  during  the  time  interval 
covered  by  the  data  record  used.  Such  an  approach  is  usually  an 
off-line  process  due  to  the  number  of  computations  required  and 
the  processing  delay. 

The  approach  considered  in  this  section  lies  somewhere 
between  the  LMS  Gradient  Algorithm  and  the  second  approach 
discussed  above.  The  Adaptive  Fixed-Point  Iteration  approach 
is  an  attempt  to  solve  an  equation  of  the  form  of  (3.26). 
However,  in  contrast  to  the  second  method  discussed  above,  both 
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the  estimates  $vv  and  R and  the  unit  sample  response  vector 
AX  “XX 

A A 

h^  are  updated  recursively.  The  estimates  and  are  up- 
dated by  the  addition  of  the  new  data  point  to  the  data  window 
being  used  while  is  updated  by  carrying  out  one  step  of  a 
fixed-point  iteration  algorithm.  Thus,  the  system  (3.26)  is 
not  solved  at  any  one  sample  time.  Rather,  the  solution  is 
obtained  iteratively  over  a number  of  samples  with  the  estimates 

/V  A 

$xx  and  being  updated  concurrently.  The  Adaptive  Fixed- 
Point  Iteration  discussed  here  is  similar  to  the  LMS  Gradient 
Method  in  that  one  step  of  an  iterative  algorithm  for  finding 
h*  is  carried  out  at  each  sample  point  in  time.  However,  it 
differs  from  the  LMS  Gradient  Algorithm  in  regards  to  the 
handling  of  the  past  data.  In  Adaptive  Fixed-Roint  Iteration, 
the  history  of  the  past  data  is  summarized  in  the  estimates 

A A 

$xx  and  R^  as  well  as  in  the  unit  sample  response  h^.  In 

constrast,  h contains  all  the  information  about  the  past  data 
— n 

available  in  the  LMS  Gradient  Algorithm.  Therefore,  Adaptive 
Fixed-Point  Iteration  lies  somewhere  between  the  two  historical 
methods  discussed  in  terms  of  computational  effort  and  data 
utilized  in  the  estimation  of  the  statistics. 


3.2.1  Fixed-Point  Iteration 

In  this  section,  the  general  case  of  fixed-point  iterative 
solutions  of  systems  of  the  form  (3.26)  will  be  discussed. 

This  will  serve  as  a framework  for  the  consideration  of  the 


particular  algorithms  discussed  in  later  sections.  Specifically, 
results  derived  for  the  general  case  can  be  directly  applied  to 
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particular  algorithms.  Initially,  it  will  be  assumed  that  the 

A A 

matrix  and  the  vector  R xx  are  constant.  The  results  de- 
rived for  this  case  will  be  extended  to  the  case  of  Adaptive 

/\ 

Fixed-Point  Iteration  where  $xx  is  a matrix  random  process  and 

A 

is  a vector  random  process . 

The  principle  behind  the  fixed-point  iterative  solution 
of  (3.26)  is  to  find  the  solution  h*  as  a limit  of  a vector 
sequence  h^,  h^,  h^ ,...  whose  terms  are  computed  recursively  as: 


^n+1 


F(h  ) 


n-0 ,1,2,... 


(3.27) 


where  hQ  is  an  initial  guess  and  the  operator  F is  chosen  in 
such  a way  that  h*  is  a fixed  point  of  F.  That  is: 

h*  * F (h*)  . (3.28) 


For  the  linear  system  (3.26)  it  is  desired  to  find  a matrix  B 
and  a vector  d so  that  the  vector  equation: 


h = B h + d 


(3.29) 


is  equivalent  to  the  system  (3.26).  That  is,  the  solution 
vector  of  (3.26)  should  also  be  a solution  of  (3.29)  and  a fixed- 
point  of  the  iteration: 


h . 
-n+1 


B h + d 
— n — 


(3.30) 
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The  matrix  B and  the  vector  d may  be  found  by  considering  the 
vector  equation : 

h = h + C-1(RXX-$XX  h)  (3.31) 

where  C is  any  invertable  NXN  matrix.  Obviously,  (3.31)  has 
a solution  if  and  only  if : 

C'1(^XX~^XX-)  = ° (3.32) 

and,  since  C was  chosen  to  be  invertable,  (3.32)  holds  if  and 
only  if: 

^XX^  = ^XX  * (3.26) 

Thus,  the  solution  of  (3.31)  is  also  the  solution  of  the  linear 

system  (3.26).  Therefore,  a fixed-point  iteration  of  the  form: 

Vi  “ + c_1Sxx  <3-33> 

may  be  used  to  find  a solution  of  the  linear  system  (3.26) 
provided  the  sequence  h^,  h^,  h2,...  produced  by  (3.33)  con- 
verges to  the  fixed-point  h*. 

To  determine  whether  or  not  the  sequence  h^  generated  by 
(3.33)  will  converge  for  all  initial  guesses  h^ , Theorem  2.1 
from  Chapter  2 can  be  applied.  Theorem  2.1  states  that  a fixed- 
point  iteration  of  the  form  (3.30)  converges  regardless  of  the 
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initial  guess  if  and  only  if  the  eigenvalues  of  the  iteration 
matrix  B are  strictly  less  than  one  in  absolute  value.  Thus, 
the  iteration  given  by  (3.33)  converges  if  and  only  if  the 
eigenvalues  of  the  NxN  matrix: 

(I-c'1Jxx» 

are  strictly  less  than  one  in  absolute  value.  Therefore,  the 

matrix  C must  be  chosen  in  such  a way  as  to  meet  this  criterion. 

In  addition  to  requiring  that  the  matrix  C be  such  that 
-1* 

the  eigenvalues  of  (I-C  4yx)  are  less  than  one  in  absolute 
value,  the  restriction  must  also  be  made  that  C be  easily 
invertable.  Obviously,  if  C is  simply  a general  NxN  matrix, 
the  iteration  would  be  pointless  since  with  the  same  amount  of 

A 

work  the  matrix  could  be  inverted  and  the  solution  found 
directly.  The  choice  of  the  matrix  C is  what  distinguishes 
particular  algorithms  within  the  class  of  fixed-point  iterative 
methods.  Two  particular  algorithms  will  be  considered  in  the 
sections  that  follow.  These  methods  use  a diagonal  matrix  and  a 
lower  triangular  matrix  respectively  for  the  matrix  C.  Such 
forms  are  easily  invertable  and  can  be  chosen  so  as  to  guarantee 
convergence  for  the  problem  considered. 

The  convergence  of  fixed-point  iteration  algorithms  can 

A 

be  guaranteed  for  a constant  matrix  4>  and  a constant  vector 

XX 

A 

if  the  matrix  C is  chosen  so  that  the  requirements  of 
Theorem  2.1  are  satisfied.  However,  in  Adaptive  Fixed-Point 


Iteration,  $xx  and  R^  are  a random  matrix  and  a random  vector 
respectively.  Thus,  the  sequence  produced  by  the  adaptive 
fixed-point  iteration  is  actually  a sequence  of  random  vectors 
and  "convergence"  of  the  algorithm  must  be  defined  as  convergence 
in  the  mean.  Thus,  the  convergence  of  the  iteration  given  by: 

ElcnW  -Bttcn-WS„,+EIW 

A /\ 

must  be  considered  where  , R™.  and  C are  the  random 

xx , n -^xx , n n 

A A 

matrix  $>xx,  the  random  vector  Rxx  and  the  random  matrix  C at 
sample  time  n.  The  matrix  Cn  is  considered  to  be  random  since 
in  the  general  case  it  may  be  derived  from  the  random  matrix 

/A 

$ v . If  the  length  M of  the  data  window  is  sufficiently  large, 
then  cross-correlation  between  terms  can  be  neglected  and  (3.34) 
cam  be  rewritten  as : 

'El6n+l!  = (I-EtCnr1El»xxn1)EtV+ElCn1'lEt£xxn1-  (3‘35) 

Now,  the  matrix  (I-E[C  1 1)  and  the  vector  E[C  1]E[RVV  1 

n xx, n n — xx,n 

are  constant  provided  that  the  process  X(n)  is  stationary.  In 

the  case  of  non-stationary  processes,  it  is  assumed  that  the 

statistics  of  X(n)  vary  slowly  with  respect  to  the  rate  of 

convergence  of  the  fixed-point  iteration  so  that  the  matrix 

E [C  ] ] and  the  vector  E [C  ] EtR^,,  ] remain  constant 

n XX , n n — xx , n 

until  the  algorithm  has  converged.  Thus,  Adaptive  Fixed-Point 
Iteration  will  converge  in  the  mean  provided  that  the  eigen- 
values of  the  matrix: 
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(I-E‘Cnrl*XX> 

are  strictly  less  than  one  in  absolute  value. 


(3.36) 


3.2.2  Selection  of  the  Matrix  C to  Insure  Convergence 

In  practice,  the  determination  of  the  eigenvalues  of  the 
matrix  in  (3.36)  may  be  difficult  since  little  or  no  a priori 
knowledge  of  the  statistics  of  the  process  X(n)  is  assumed. 

Thus,  it  is  useful  to  have  some  bound  on  the  eigenvalues  of 
the  matrix  in  (3.36).  If  X is  an  eigenvalue  of  the  matrix  in 
(3.36),  then  the  following  equation  is  satisfied: 

DET [XI- (I-E [Cl _1$) ] = 0 
n xx 

or: 

DET[E[Cn]_1( (X-l)E[Cn]+$xx) ] = 0 . (3.37) 

The  determinant  of  a product  is  the  product  of  the  determinants. 
Thus,  (3.37)  can  be  rewritten  as: 

DET [E [C  ]_1]DET[ ( X-l) E [C  ] +$VY]  = 0 . 

n n aa 

But,  by  hypotheses,  E [Cn ] 1 is  invertable  and  thus  DET[E[Cn]  1] 

^ 0.  Therefore, 

DET[  (X-l)  E [C  ]+4>  v]  = 0 . (3.38) 

n Xa 
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From  (3.38),  it  follows  that  the  matrix: 

(I-1)E[CnI+*XX 

is  singular.  Thus,  by  Hadammard's  Theorem  [19]: 

' 

N 

| (X-1)E[C.  •]+<{>..  | _<  I | (X-l)C.  .+<J>.  . | (3.39) 

11  j iJ 

j*i 

where  4> . . and  C.  . are  the  i,j  elements  of  the  matrices  $v 
and  Cn  respectively. 

Equation  (3.39)  does  not  present  a simple  expression  for 

a bound  on  the  values  of  the  eigenvalues  of  the  matrix  in  (3.36). 

However,  as  will  be  seen  in  later  sections,  (3, 39)  will  provide 

a useful  bound  for  the  particular  algorithms  considered  in  this 

dissertation  and  will  allow  the  selection  of  the  matrix  C so 

n 

that  convergence  in  the  mean  is  guaranteed. 

3.2.3  Rate  of  Convergence  of  the  Mean  Square  Error 

Once  it  has  been  determined  by  the  use  of  Theorem  2.1 
that  the  vector  sequence  E[hg],  E[h^],...  produced  by  Adaptive 
Fixed-Point  Iteration  converges  to  the  desired  solution  h * , 
it  is  then  of  interest  to  consider  the  rate  of  convergence  of 
the  sequence.  As  in  the  case  of  the  LMS  Gradient  Algorithm, 
it  is  of  the  most  interest  to  consider  the  convergence  of  the 
mean-square  error  produced  by  the  Adaptive  Fixed-Point  Itera- 
tion to  the  minimum  mean-square  error  produced  by  the  optimal 
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filter.  The  mean-square  error  produced  by  the  Adaptive  Fixed- 
Point  Iteration  Algorithm  at  any  sample  time  n is  given  by: 
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MSE(n)  » E[  (S (n)-S (n) ) 2]  - E [S2  (n)  ] -2E  [S  (n)  S (n)  ] +E  [£2  (n)  ] . 

(3.40) 

Substituting  the  filter  output  equation  (1.7)  into  (3.40)  gives: 

MSE(n)  - E[S2  (n)  ]-2E[S(n)XT(n-n1  )h  ] + E [h^X (n-n. ) XT (n-n. ) h ] . 

If  cross-correlations  between  terms  are  neglected,  the  above 
expression  can  be  written  as: 

MSE(n)  * E[S2(n)]-2E[S(n)XT(n-n1)]E[h]+E(hT)E[X(n-n1)XT(n-n, ) ] 

— i — n — i — i 

E[h  ] . (3.41) 

— n 

Finally,  applying  definitions  (1.11)  and  (1.12)  to  (3.41) 
yields: 

MSE(n)  = E[S2(n) ]-2R^y  E [hi +E [h?] $yyE [h  ] . (3.42) 

— b X — n — n xx  — n 


1 


Equation  (3.38)  then  gives  an  approximation  to  the  mean-square 
error  produced  by  the  Adaptive  Fixed-Point  Iteration  algorithm 
where  the  sequence  Eth^]  is  produced  by  the  recursive  relation- 
ship (3.34)  . 

The  mean-square  error  produced  by  the  optimal  filter  is 
given  by  (2.33)  to  be: 
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MSEMn)  = E [S2  (n)  J - 2 R^x  h*  + h*T  $xx  h*.  (2.33) 

Thus,  the  difference  between  the  estimate  of  the  mean -square 
error  MSE(n)  produced  by  the  Adaptive  Fixed-Point  Iteration 
Algorithm  and  the  minimum  mean-square  error  MSE*(n)  produced  by 
the  optimal  filter  h*  can  be  found  by  subtracting  (2.33)  from 
(3.42)  which  gives: 

MSE  (n) -MSE*  (n)  = E [h£]  ^E  [hfi]h*T*xxh*-2RgX [E  -h*]  . (3.43) 

Thus , 

MSE (n) -MSE* (n)  * (E [hj] -h*T) $VY (E [h  ] +h*) -2R^V [E (h  ] -h*] 

— n — xx  — n — — 5X  — n — 

or: 

MSE (n) -MSE* (n)  = (E [hj] -h*) [$YYE [h  ] +$YYh*-2RCY]  . (3.44) 

- n — xx  — n xx—  — sx 

But,  from  (1.14)  Rgx  may  be  written  as: 

-SX  " $XX^*  ' (3.45) 

Thus,  by  substituting  (3.45)  into  (3.44),  (3.44)  may  be  rewritten 
as : 

MSE (n) -MSE* (n)  = (E [h  ] -h*) T$  (E [hi -h*) . 

n — XX  — n — 


(3.46) 
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Thus,  (3.46)  gives  a bound  on  the  rate  of  convergence  of  the 
mean-square  error  produced  by  the  adaptive  fixed-point  iteration 
algorithm.  This  bound  may  be  generated  for  increasing  n by 
recursively  generating  the  vector  sequence: 


Y - E[h  ]-h*  (3.47) 

— n — n 


and  using  the  resulting  values  in  (3.46)  to  compute 

(MSE (n) -MSE* (n) ) . Since  h*  is  assumed  to  be  a fixed  point  of 

the  iteration  (3.35),  h*  must  satisfy: 


h*  - ( I-E tcn! _1*xx) h*+E IChI'1^ 


(3.48) 


Thus,  by  subtracting  (3.48)  from  (3.35)  the  following  recursive 
relationship  for  the  vector  sequence  may  be  obtained: 


Vl 


<I-E'cn]'1«xx)Xn 


(3.49) 


In  the  case  of  the  LMS  Gradient  Algorithm,  a simpler 
expression  for  the  difference  (MSE (n) -MSE* (n) ) was  obtained  by 
uncoupling  the  variable  h^.  This  could  be  accomplished  because 
the  matrix  (I-2y  4^)  is  real  and  symmetric  and  therefore  has 
distinct  eigenvalues.  Thus,  the  matrix  (I-2y  *xx)  could  be 
diagonalized.  However,  in  the  general  case  of  fixed-point 
iteration  considered  here  the  matrix  (I-E[Cn]  may  or  may 

not  have  distinct  eigenvalues.  But,  the  matrix  4*^  is  real  and 
symmetric  and  thus  has  distinct  eigenvalues.  Therefore,  if  the 


, ..  . ' ■ V.  - '-.'J--  ....  - ' LV—1  ■ . 


(3.50) 


new  variables  h and  h*  are  defined  as: 
— n — 


h=  M-1  E[h  ] = MT  E [h  ] 
— n — n — n 


ii*  = M_1  h*  = MT  h* 


(3.51) 


where  M is  the  modal  matrix  of  $ , then  in  terms  of  the  new 

AA 


variables  and  h*,  expression  (3.46)  may  be  rewritten  as: 


MSE  (n)  -MSE*  (n)  * (Mfh^-h*)  ) T*xx**(h^-h*) 


or: 


MSE  (n) -MSE*  (n)  = (h^-h*)  TMT$XXM  (^-h*) 


(3.52) 


Thus,  since  Mi$xxM  is  the  diagonal  eigenvalue  matrix  of  $xx, 
(3.52)  can  be  expressed  as: 


MSE (n) -MSE* (n)  = (h  -h*) TA  (h  -h*) 

— n — — n — 


(3.53) 


or: 


N 


MSE (n) -MSE* (n)  = Z (h  (i) -h* (i) ) 

i-1  1 n 


(3.54) 


The  vector  sequence  (h^h*)  may  be  generated  recursively  by  the 


following  formula  derived  from  (3.49)  where  Y^  is  defined  as 
Y = (h  -h*) . Thus,  from  (3.49): 
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2n+l 


(I-MTE[Cn]"1<txxM)Y 


— n 


(3.55) 


or: 


n+1 


(I-MTE[C  ]_1MM  Y, 
n — r 


(3.56) 


In  general,  the  equations  (3.53)  and  (3.56)  can  be  used 
to  determine  a bound  on  the  rate  of  convergence  of  Adaptive 
Fixed-Point  Iteration  Algorithms.  However,  in  the  special  case 
where  the  matrix: 


D = MT  E[C  j"1  M 
n 


is  diagonal,  then  the  variables  ^ will  be  uncoupled  and  (3.54) 


can  be  rewritten  as: 


N 


MSE (n) -MSE* (n)  » Z X (1-X  ) 2n (h  (i) -h* (i) ) 2 

i=l  1 1 0 


(3.57) 


•*  1.U  M 1 

where  X.  is  the  i eigenvalue  of  the  matrix  E[C  ] $vv. 
x n aa 

Expression  (3.57)  is  identical  to  (2.49)  (which  was  derived  for 
the  LMS  Gradient  Method)  with  X^  replaced  by  2 y X^  Thus,  in 
regards  to  convergence  in  the  mean,  the  LMS  Gradient  Algorithm 
behaves  like  a fixed-point  iteration  with  the  C matrix  chosen  as : 


C-1  = 2 ul  . 


(3.58) 


Equations  (3.54),  (3.55)  and  (3.57)  are  the  fundamental 
results  of  this  section.  In  the  case  of  fixed-point  iteration 
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with  an  arbitrary  matrix  Cn  equations  (3.54)  and  (3.55)  can  be 
used  to  investigate  the  rate  of  convergence  in  the  mean  of 
Adaptive  Fixed-Point  Iteration.  Equation  (3.57)  may  be  used  in 
place  of  equations  (3.54)  and  (3.55)  in  the  special  case  where 
the  modal  matrix  of  $xx  will  diagonalize  the  iteration  matrix 
(I-E  [C  ] 1<t») . Specifically,  these  expressions  allow  the 
computation  of  a lower  bound  on  the  mean-square  error  produced 
by  the  filter  determined  by  Adaptive  Fixed-Point  Iteration  as 
a function  of  n.  Equations  (3.54)  and  (3.55)  and  equation  (3.57) 
give  a lower  bound  in  that  convergence  will  proceed  according 
to  these  relationships  only  if  the  statistics  $xx  and  are 
known  exactly.  While  the  estimates  $xx  and  Rxx  are  consistent 
estimates  of  4>xx  and  Rxx  respectively,  they  do  have  finite 
variance  when  the  data  window  used  in  computing  them  is  of 
finite  duration  as  in  the  case  in  Adaptive  Fixed-Point  Itera- 
tion. However,  while  only  an  approximation,  the  bound  computed 
using  (3.54)  and  (3.55)  or  by  using  (3.57)  is  useful  for  com- 
paring the  rates  of  convergence  of  fixed-point  iteration  algor- 
ithms and  the  LMS  Gradient  Method.  Such  comparisons  will  be  made 
in  Chapter  V while  the  relationship  between  the  theoretical 
and  actual  convergence  properties  of  Adaptive  Fixed-Point  Iter- 
ation Algorithms  and  the  LMS  Gradient  Method  will  be  explored 
via  computer  simulation  in  Chapter  VI. 


• 3M 


I 


' ! 
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3.3  The  Jacobi  Relaxation  Method  for  Adaptive  Fixed-Point 
Iteration 

A general  form  for  Adaptive  Fixed-Point  Iteration  Algor- 
ithms was  given  by  (3.33)  as: 


^n+1  " (I_Cn  *XX,n,^n+Cn  ^XX,n 


In  order  for  an  Adaptive  Fixed-Point  Iteration  Algorithm  to  be 

implemented,  some  method  for  determining  the  matrix  CR  must  be 

given.  As  stated  previously,  the  restrictions  on  the  choice  of 

the  matrix  C are  that  C be  easily  invertable  and  that  all  the 
n n 

eigenvalues  of: 


(I"E[cn,l*xX) 


be  less  than  one  in  absolute  value.  The  Jacobi  Relaxation 
Method  attempts  to  satisfy  both  these  restrictions  using  a 
diagonal  matrix  for  Cn-  Obviously,  a diagonal  matrix  is  easily 
invertable  and  the  particular  diagonal  matrix  used  in  Jacobi 
Relaxation  can  be  chosen  so  as  to  satisfy  the  constraints  on  the 
eigenvalues  of  the  matrix  (I-E[CnJ  1$xx) . 

The  matrix  $vv  is  symmetric  and  may  be  written  as  the 


XX, n 


= L +D  +L 

Xa  f n 


(3.59) 
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where  Ln  is  a strictly  lower  triangular  matrix  and  Dr  is  a 

diagonal  matrix.  . In  the  Jacobi  Relaxation  Method  the  matrix 

C is  chosen  to  be: 
n 

C = 1/8  D (3.60) 

n n 

where  the  constant  parameter  8 > 0 is  referred  to  as  the  relax- 
ation parameter.  Thus,  the  Jacobi  Relaxation  Method  is  defined 
as: 


Vl  - (I-e  Dn1  JXX,n>!!n  + 6 Dn 


~X  Sxx.n 


(3.61) 


Therefore,  by  the  results  of  the  preceeding  section,  the  Jacobi 
Relaxation  Method  will  converge  if  and  only  if  the  eigenvalues 
of  the  matrix: 


(1-8  E [ Dn ] ~ 1 $xx)  (3.62 

are  less  than  one  in  absolute  value.  But,  since  it  was  assumed 
that  the  process  X(n)  was  stationary  over  the  length  of  the 
filter,  the  elements  on  the  diagonal  of  the  matrix  Dn  are 
identical.  That  is: 


E‘dii,n’  ’ Rxx(0)  ' 


(3.63) 


Thus,  the  matrix  in  (3.62)  may  be  rewritten  as: 


■ ■ ■. 
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(I  " 6 rJ^ToT  $XX}  ‘ 


(3.64) 


The  matrix  in  (3.54)  is  of  the  form  (I-kA)  and,  as  was  shown 
in  Chapter  2,  the  eigenvalues  of  this  form  are  given  by: 


Xj_  « (1-K  Xi)  i*0 , 1 , . . . ,N-1 


where  X^  are  the  eigenvalues  of  the  matrix  A.  Thus,  the  eigen- 
values of  the  matrix  (3.64)  are  giver,  by: 


- (1  - 


X^  i*0 ,1, . . . ,N-1 


(3.65) 


where  X^  are  the  eigenvalues  of  the  autocorrelation  matrix  $xx. 
Since  the  restrictions  on  the  eigenvalues  of  the  iteration 
matrix  require  that: 


1 - S^ToT  XJ  <:1  i”0'1 "-1 


and  8 was  assumed  positive  and  all  the  X^  are  positive,  the 
Jacobi  Relaxation  Method  will  converge  provided  that: 


0 <6  <2'\nax  • BXX(0> 


(3.66) 


where  X is  the  largest  eigenvalue  of  $ v. 
max  xx 
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3.3.1  Selection  of  0 to  Insure  Convergence 

Equation  (3.66)  gives  a relationship  which  allows  the 
selection  of  the  relaxation  parameter  8 to  insure  convergence 
of  the  Jacobi"  Relaxation  Method.  However,  application  of 
(3.66)  requires  a priori  knowledge  of  the  largest  eigenvalue 
of  <I>XX  and  R^fO).  This  information  may  not  be  available  in 
the  general  case  where  little  or  no  a priori  knowledge  of  the 
statistics  of  the  process  X(n)  is  assumed.  In  this  case,  a 
value  for  0 which  will  guarantee  convergence  can  be  obtained 
by  using  the  results  of  Section  3.3.2  to  obtain  a bound  on 
A which  can  then  be  used  in  (3.66)  to  determine  an  appropriate 
value  of  0.  Equation  (3.39)  states  that  if  A is  an  eigenvalue 
of  the  matrix: 

then  A satisfies  the  following  inequality: 

N 

| (A-l)EtC.  .]+<}>..  | >_<  2 | (A-1)C..+<|>.  . | . (3.39) 

j=l 

In  the  Jacobi  Relaxation  Method,  the  matrix  Cn  is  the  diagonal 

A 

matrix  whose  diagonal  is  the  same  as  that  of  the  matrix 

XX,  n 

multiplied  by  1/0.  Thus, 
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Therefore,  (3.39)  can  be  written  for  the  case  of  Jacobi  Relax- 
ation as : 


N 


I (X-l)  j _<  ^ I ♦j.  j I 


j=l 


(3.67) 


In  the  case  of  a stationary  process  X(n),  the  elements  of  the 
matrix  4>xx  are  given  by: 


^ij  Rxx*i“^ 


Thus,  (3.67)  can  be  rewritten  as: 


N*-l>  • I Rxxl0)+Rxx(0)l 


|Rxx(1‘3) 

j^l 


(3.68) 


If  X is  an  eigenvalue  of  the  matrix  in  (3.64)  , then: 


(I  - 


*XX 


TOY  WI-  x I 


where  ¥ is  the  eigenvector  corresponding  to  X < Thus : 


* w 

xxl 


*xx(0) 


(X-l) ¥ 


Rxx (0)  - 

Therefore,  the  constant  — g (X-l)  is  an  eigenvalue  of  the 

autocorrelation  matrix  $xx»  With  this  substitution,  (3.68) 
can  be  expressed  as: 
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i 


lX+RXX(0)l  - E lRxx(i"j) I * (3.69) 

j=l 

Now,  since  ^ is  an  eigenvalue  of  the  positive  definite  matrix 

2 

and  Rxx  ( 0 ) =■  E[X^nj],  both  X and  Rxx(°)  are  greater  than 
zero.  Thus,  from  (3.69): 

N 

X.J  Z | RxX ( i- j ) | - RxX(0)  . (3.70) 

j^l 

But,  since  the  autocorrelation  function  has  a maximum  at  zero, 
the  following  inequality  can  be  obtained  from  (3.70): 

Xmax  < <a-71> 

where  N^^-n^+l)  is  the  length  of  the  filter.  Thus,  substi- 
tuting (3.71)  into  (3.66)  the  following  bound  on  the  value  of 
6 which  will  insure  convergence  is  obtained: 

0.  < 0 < • (3.72) 

The  bound  given  by  (3.72)  allows  the  selection  of  a value  for 
the  relaxation  parameter  0 which  will  guarantee  convergence 
of  the  Jacobi  Relaxation  Algorithm  regardless  of  the  values  of 
the  eigenvalues  of  the  autocorrelation  matrix.  However,  the 
selection  of  a value  for  0 which  is  greater  than  (2/N-2)  may 
also  cause  the  algorithm  to  converge  and  the  corresponding  rate 
of  convergence  will  be  more  favorable. 
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3.3.2  Rate  of  Convergence  of  the  Mean-Square  Error 

It  is  obvious  from  equation  (3.66)  that  the  parameter  8 
must  be  chosen  within  a certain  range  to  insure  convergence. 
However,  the  value  of  6 chosen  also  effects  the  rate  of  con- 
vergence of  the  algorithm.  Specifically,  the  convergence  of 
Jacobi  Relaxation  can  be  studied  by  the  use  of  equations  (3.54) 
and  (3.55)  which  were  derived  in  Section  3.2.3.  From  these 
equations,  it  can  be  seen  that  for  the  Jacobi  Relaxation  Method, 
a bound  on  the  rate  of  convergence  is : 

N „2 

MSE  (n)  -MSE*  (n)  -EX.  y (i)  (3.73) 

i-1  1 n 

where : 


2n+l 


(I  - 


8 

55*n- 


A) 


Y 

-n 


(3.74) 


and  Xi  and  M are  the  eigenvalues  and  the  modal  matrix  respect- 
ively of  the  autocorrelation  matrix  *xx- 

The  approximate  bound  can  also  be  obtained  by  recognizing 
that  in  this  special  case,  the  modal  matrix  M diagonalizes  the 
iteration  matrix.  Thus,  equation  (3.57)  yields: 


MSE  (n)  -MSE*  (n)  = E^  X±(l  - r^\o)  V 2n  (hQ  (i)  -h*  (i)  ) 2 . (3.75) 

where  h^  and  h*  are  the  uncoupled  initial  unit-sample  response 
vector  and  the  optimal  uncoupled  unit-sample  response  vector, 
respectively.  Thus,  the  convergence  of  Jacobi  Relaxation  may  be 
studied  more  easily  by  the  alternate  form  (3.75).  The  rate  of 


convergence  for  various  values  of  the  relaxation  parameter  6 
and  the  relationship  of  the  rate  of  convergence  of  the  LMS 
Gradient  Algorithm  will  be  considered  in  Chapter  V. 
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3.3.3  Alternate  Form  for  the  Jacobi  Relaxation  Method 


The  form  for  Jacobi  Relaxation  given  in  (3.6.1)  is  useful 
when  considering  the  algorithm  as  an  example  of  the  general 
fixed-point  iteration  method  considered  in  Section  3.2. 
Specifically,  using  the  form  of  (3.61)  allowed  the  application 
of  the  results  on  convergence  derived  in  Section  3.2  to  the 
case  of  the  Jacobi  Relaxation  Method.  However,  when  the 
implementation  of  the  algorithm  is  considered,  an  alternate  form 
for  the  Jacobi  Relaxation  Method  is  useful.  In  addition,  the 
alternate  form  allows  a more  intuitive  approach  to  understanding 
the  algorithm. 

The  alternate  form  is  obtained  by  considering  the  ith 

component  h.  of  the  unit  sample  response  vector  h . From  (3.61), 
i — n 

hi,n+l  is  given  by: 


hi,n+l  = hi,n 


‘ 6 


N „ 
E 


h. 


d> . . j=«l 
Tn,n  J 


13, n j ,n 


(p  . . 
^ii  ,n 


R.  _ i=l,2,...N. 

i,n 

(3.76) 


~ ~ -hh  a 

where  <J>.  . and  R.  are  the  (i,j)  element  of  <J>VV  and  the 
i j r n l f n A a f n 

th  * „ 

i component  of  Rxx  n respectively.  The  alternate  form  (3.76) 

may  be  more  useful  for  the  implementation  of  the  algorithm  than 

the  vector  form  (3.61)  since  the  matrix  multiplications  have 

been  broken  down  into  scalar  multiplications  and  additions. 


«l 


1 


J 
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In  the  special  case  where  the  relaxation  parameter  B 
equals  one,  equations  (3.61)  and  (3.76)  are  simply  alternate 
forms  for  the  familiar  Jacobi  Iteration  Algorithm.  In  the 
case  where  6*1,  the  algorithm  can  be  easily  explained.  Starting 
with  an  initial  "guess"  h^  for  the  solution,  the  first  equation 
of  the  linear  system  is  solved  for  the  first  component  of  the 


vector  h.  A value  for  this  component  is  computed  by  substitut- 
ing the  components  of  the  vector  for  the  corresponding 
components  of  h on  the  right-hand  side  of  the  resulting  equa- 
tion. This  value  then  becomes  the  first  component  of  the 
vector  hjj+i*  Each  equation  of  the  linear  system  in  turn  is 
treated  in  the  same  fashion.  That  is,  the  i equation  is 
solved  for  the  ifc^  component  of  the  vector  h and  a value  for 
the  ith  component  of  ^+1  is  computed  by  substituting  the 
necessary  components  of  the  vector  for  the  corresponding 
components  of  h on  the  right-hand  side  of  the  resulting  equation. 
Once  all  N components  of  h^+1  are  obtained  one  step  of  the  iter- 
ation has  been  completed.  Equation  (3.76)  with  B=1  is  the 
relationship  required  to  carry  out  the  process  just  described. 

Another  way  of  viewing  this  algorithm  is  to  note  that  the 
procedure  (with  3=1)  finds  the  ith  component  of  the  vector  hR+1 

i.U 

in  a way  which  would  make  the  i component  of  the  residual 
vector : 


rn  ” 5xx,n  ~ $XX,n 


(3.77) 


equal  to  zero  if  the  i component  of  h^  was  replaced  by  the  i 
component  of  h^^. 


However,  if  B^l  is  used  in  (3.7'6),  then  the 
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ith  component  of  h^^  is  found  in  a way  which  would  make  the 
corresponding  component  of  r^  not  equal  to  zero  if  the  ith 
component  of  h^^  is  substituted  for  the  i^*1  component  of  h^. 
Specifically,  the  ifc^  component  of  computed  using  (3.77) 
with  the  ith  component  of  h^  replaced  by  the  ifc^  component  of 
hfi+^  will  be  less  than  zero  if  0 > 1 and  greater  than  zero  if 
0\  <1.  Thus,  with  0 > 1 the  algorithm  is  overcorrecting  or 

I 

"overrelaxing"  and  with  0 < 1 the  algorithm  is  undercorrecting 

or  "underrelaxing".  If  the  iteration  "converges",  all  components 

of  the  residual  vector  r will  be  essentially  equal  to  zero. 

Obviously,  r cannot  be  made  exactly  equal  to  zero  in  all  cases 
—n 

with  finite  precision  arithmetic.  Thus,  intuitively  it  can  be 
seen  that  the  progress  that  Jacobi  Relaxation  makes  toward  the 
solution  vector  at  each  iteration  step  is  directly  dependent  on 
the  value  chosen  for  0.  This  conclusion  is  confirmed  by  the 
discussion  on  the  rate  of  convergence  of  the  algorithm  presented 
in  the  last  section. 

The  Jacobi  Relaxation  Method  is  one  example  of  a fixed- 
point  iterative  algorithm  which  can  be  applied  to  the  adaptive 
estimation  problem.  It  has  been  shown  that  the  algorithm  con- 
verges provided  that  certain  restrictions  on  the  value  of  the 
relaxation  parameter  0 are  met.  A bound  on  the  value  of  0 
which  will  insure  convergence  of  the  algorithm  was  developed. 

This  bound  is  independent  of  the  statistics  of  the  process. 

In  addition,  a bound  on  the  rate  of  convergence  of  the  algorithm 
is  given.  A comparison  of  the  theoretical  bounds  on  the  rates 
of  convergence  of  the  Jacobi  Relaxation  Method,  the  LMS  Gradient 
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Method  and  the  Successive  Over-Relaxation  Method  presented  in 
the  next  section  will  be  given  in  Chapter  V.  The  relationship 
between  the  theoretical  bounds  and  actual  performance  will  be 
explored  via  computer  simulation  in  Chapter  VI. 


3.4  The  Successive  Over-Relaxation  Method  for  Adaptive  Fixed- 
Point  Iteration 


The  Jacobi  Relaxation  Algorithm  was  obtained  by  the  choice 
of  a particular  diagonal  matrix  given  by  (3.60)  for  the  CR  in 
the  general  Adaptive  Fixed-Point  Iteration  formula  (3.33).  The 
diagonal  form  is  easily  invertable  and  the  particular  matrix 
chosen  insured  the  convergence  of  the  algorithm  provided  that 
the  relaxation  parameter  8 was  chosen  to  be  within  a certain 
range.  In  the  Successive  Over-Relaxation  Method  (SOR) , the 
matrix  Cn  is  chosen  to  be  the  sum  of  a diagonal  matrix  and  a 
lower  triangular  matrix.  This  form  is  also  easily  invertable 
and  can  be  chosen  so  as  to  insure  the  convergence  of  the  algor- 
ithm. That  is,  a matrix  C of  this  form  can  be  easily  chosen  so 

n 

that  the  eigenvalues  of  the  matrix: 


1 


(I-EtCnrl  W 


(3.78) 


are  strictly  less  than  one  in  absolute  value. 

A 

The  symmetric  matrix  $xx  n can  be  decomposed  as  in  (3.59) 


into  the  sum: 
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$ 

XX,  n 


L + D 
n n 


(3.59) 


where  is  a strictly  lower  triangular  matrix  and  is  a 

diagonal  matrix.  In  the  SOR  method,  the  matrix  C is  chosen  to 

n 

be: 

Cn  ■ I D„  + Ln  (3'75» 

where  the  constant  parameter  B > 0 is  referred  to  as  the  relax- 
ation parameter.  Thus,  the  SOR  method  for  Adaptive  Fixed-Point 
Iteration  is  defined  as: 


^l+l 


(I-B(D  +BL  j"1*  )h +6(D  +6L  ) 
n n XX, n — n n n 


-1 


^XX,n‘ 


(3.80) 


Therefore,  by  the  results  of  Section  3.2.1,  the  SOR  method  will 
converge  if  and  only  if  the  eigenvalues  of  the  matrix: 


(I-BEID  +BL  ]_1$YY) 
n n XX 


(3.81) 


are  strictly  less  than  one  in  absolute  value.  However,  it  is 
difficult  to  determine  the  eigenvalues  of  the  matrix  in  (3.81). 
Fortunately,  it  is  possible  to  prove  that  all  the  eigenvalues 
of  the  iteration  matrix  are  less  than  one  in  absolute  value  for 
a range  of  the  relaxation  parameter  8 which  does  not  depend  on 
the  eigenvalues.  In  order  to  prove  the  convergence  of  the  SOR 
method,  the  following  theorem  is  needed.  A proof  of  the  theorem 
can  be  found  in  [20]. 


j 

P 
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Theorem  3 . 1 

Let  A be  a real  matrix.  Then  all  the  eigenvalues  of  A 
are  strictly  less  than  one  in  absolute  value  if  and  only  if  (I 
is  non-singular  and  there  exists  a real  positive  definite 
matrix  Q such  that: 


P - HQ  + QHT  (3.82) 

is  positive  definite,  where: 

H = (I-A)-1 (I+A)  . (3.83) 

Applying  Theorem  3.1  to  the  SOR  Method,  the  matrix  A is 
given  by: 


A * (I-BE[D  +BL  l”1*—.)  . 

n n aa 

Thus,  the  matrices  (I-A)  and  (I+A)  are: 

(I-A)  * 8E  [D  +BL  ] _1<fYV  (3.84) 

n n xx 

(I+A)  = 2I-SE [D  +BL  ] -1$vv  . (3.85) 

n n xx 

Obviously,  the  matrix  (I-A)  as  given  in  (3.84)  is  non-singular 

since  both  E[D  +8L  ] 1 and  are  non-singular.  Thus,  the 
n n xx 

matrix  H * (I-A)  ^(I+A)  for  the  SOR  Method  is: 


■A) 
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E>VeLn)(2I-8E[VSLn1'\x) 


or,  equivalently: 


-1 

H * 2/8  $YY  E [D  +6L  ] - I 
AA  n n 


H = 2/6  [E [D  ] +6E [L  ] -6/2  $YY] 
xx  n n xx 


Thus , H can  be  written  as : 


H = Cj  E[D  ]-E[lJ]+E|L  ] ] . (3.86) 

xx  p n n n 

Now,  a positive  definite  matrix  Q must  be  found  so  that  the 
matrix  in  (3.83)  is  positive  definite.  The  autocorrelation 
matrix  is  positive  definite  by  Theorem  1.1.  Thus,  its  inverse 
is  also  positive  definite.  If  is  chosen  for  the  matrix 
Q,  then  the  expression  in  (3.83)  may  be  rewritten  as: 

P = H ^ HT  . (3.87) 

Substituting  from  (3.86)  for  H in  (3.87)  yields: 

P=^[¥E[Dn'-E[L^+E[Ln'l*XX+4XX[¥EtDn1-E[Ln'T+E'VT'^X- 

(3.88) 

But,  E[LT]T=E[L  ].  Thus,  (3.88)  can  be  rewritten  as: 
n n 


P * 2 ‘¥>  *3  ElDn>*'xx 


(3.89) 


which  is  positive  definite  for  any  value  of  8 in  the  range: 


0 < 6 < 2 . 


(3.90) 


Thus,  by  Theorem  3.1,  all  the  eigenvalues  of  the  matrix 
(3.81)  are  strictly  less  them  one  in  absolute  value  provided 
that  the  restrictions  on  8 given  by  (3.90)  are  met.  Thus,  by 
Theorem  2.1,  the  SOR  Method  will  converge  for  any*  value  of  the 
relaxation  parameter  between  0 and  2.  Therefore,  no  knowledge 
of  the  statistics  of  the  process  is  required  to  determine  a 
value  for  8 which  will  insure  convergence  of  the  SOR  Method. 
However,  the  selection  of  the  value  of  6 has  an  important  effect 
on  the  rate  of  convergence  of  the  SOR  Method  as  will  be  shown  in 
the  next  section. 


3.4.1  Rate  of  Convergence  of  the  Mean-Square  Error 

In  the  previous  section,  it  was  shown  that  the  SOR  Method 
converges  for  any  value  of  the  relaxation  parameter  8 such  that 
0 < 8.  <2.  However,  the  value  of  8 also  effects  the  rate  of 
convergence,  of  the  algorithm.  The  convergence  of  the  SOR  Method 
can  be  studied  by  the  use  of  equations  (3.54)  and  (3.55)  from 
Section  3.2.3.  From  these  equations,  it  can  be  seen  that  for 
the  SOR  Method,  a bound  on  the  rate  of  convergence  can  be 
determined  from: 


• V, 
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and : 


N _2 

MSE(n) -MSE* (n)  = I X.  y (i) 

i=l  1 n 


£n+i  = <i-MTSE[Dn+Bi,nr1Mft)  y, 


(3.91) 


(3.92) 


where  X^  and  M are  the  eigenvalues  and  modal  matrix  respectively 

of  the  autocorrelation  matrix  $xx*  Now,  in  general,  the  modal 

matrix  M of  v will  not  diagonalize  the  matrix  E[D  + BL  ] 
aa  n n 

Thus,  the  alternate  form  for  the  rate  of  convergence  given  by 
(3.57)  cannot  be  used  to  study  the  convergence  of  the  SOR 
Method.  However,  equations  (3.91)  and  (3.92)  can  be  used  to 
generate  the  bound  sequentially  even  though  the  amount  of  com- 
putational effort  required  limits  the  usefulness  of  this  ap- 
proach when  the  length  of  the  filter  is  large.  The  rate  of 
convergence  of  the  SOR  Method  for  various  values  of  fi  and  its 
relationship  to  the  rates  of  convergence  of  the  LMS  Gradient 
Algorithm  and  the  Jacobi  Relaxation  Method  will  be  considered 
in  Chapter  V. 


i 


3.4.2  Alternate  Form  for  the  Successive  Over-Relaxation  Method 


The  SOR  Method  for  Adaptive  Fixed-Point  Iteration  was 
given  in  the  general  form  for  fixed-point  iteration  in  (3.80). 
The  use  of  this  vector  form  allows  the  SOR  Method  to  be  fitted 
into  the  general  theory  of  fixed-point  iteration.  Specifically , 
use  of  (3.80)  allows  the  application  of  the  results  on  con- 
vergence in  Section  3.2  to  the  SOR  Method.  However,  when  the 
implementation  of  the  SOR  Method  is  considered,  an  alternate 
form  for  the  algorithm  may  be  useful.  The  alternate  form  is 
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obtained  by  considering  the  i^  component  h.  of  the  unit-sample 

i / n 

response  vector  1^.  From  (3.80),  hn+1  may  be  written  as: 


Sb+1  “ (1-8»!!n-8Dnll'„!!„*l-6D;lLnV6Dn1Sxx,n 


Thus,  the  i component  n+^  of  the  vector  hn+1  is  given  by: 

hi,n+l  * {1"6)hi,n  “ j hi ,n+l^i j ,n+ ._?+.hi ^^ij ,n  1 

^ii,n  J J 


$ . . 

Tn,n 


R. 

i/n 


(3.93) 


, • th 


where  4> . . and  R.  are  the  i,^*  element  of  the  matrix  *vv 

l] ,n  i # n xx  t n 

fh  a 

and  the  l component  of  the  vector  Rxx  n respectively.  Since 
the  matrix  products  have  been  broken  down  into  scalar  multi- 
plications and  additions,  the  alternate  form  (3.93)  may  be  more 
useful  for  implementation  than  the  vector  form  (3.80)  for  the 
SOR  Method.  In  addition,  the  alternate  form  allows  a more 
intuitive  approach  to  understanding  the  algorithm. 

In  the  special  case  where  the  relaxation  parameter  8 
equals  one,  equations  (3.80)  and  (3.93)  are  simply  alternate 
forms  for  the  well-known  Gauss-Seidel  Iteration  Algorithm.  Thus, 
in  the  case  where  8=1,  the  algorithm  can  be  easily  explained  and 
is  in  fact  similar  to  the  Jacobi  Iteration  Algorithm  except 
that  each  new  element  of  the  unit-sample  response  vector  im- 
mediately replaces  h.  and  is  used  in  computing  the  remaining 

i f n 

elements  of  h^^.  Therefore,  starting  with  an  initial  "guess" 


XX, n 
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h for  the  solution,  the  first  equation  is  solved  for  the  first 
— n 


component  of  the  vector  h.  A value  for  this  component  is  then 
computed  by  substituting  the  components  of  the  vector  h^  for 
the  corresponding  components  of  h on  the  right-hand  side  of  the 
resulting  equation.  This  value  then  becomes  the  first  component 


of  the  vector  hn+1*  The  remaining  N-l  equations  are  then 


. th 


treated  in  the  following  manner.  The  i equation  is  solved 


till 

for  the  i component  of  h and  a value  for  the  i component  of 


hn+^  is  computed  by  substituting  the  first  (i-1)  elements  of 
^n+1  and  the  last  (N-i)  elements  of  h^  for  the  corresponding 
elements  of  h on  the  right-hand  side  of  the  resulting  equation. 
Once  all  N components  of  hft+1  have  been  computed,  one  step  of 
the  iteration  has  been  completed.  Equation  (3.92)  with  3=1  is 
the  recursive  relationship  for  carrying  out  the  process  just 
described. 


The  SOR  algorithm  with  3=1  can  be  viewed  in  another  way 


. th 


by  noting  that  the  i component  of  the  vector  h^^  is  chosen 

. th 


in  such  a way  as  to  make  the  i component  of  the  residual  vec- 
tor 


^XX,n  “ $XX,n  ^n 


(3.77) 


equal  to  zero  if  the  first  i components  of  were  replaced  by 


the  first  i components  of  h^+^*  However,  if  3^1  is  used  for 

. th 


the  relaxation  parameter,  then  the  i component  of  hn+1  is 
found  in  a way  which  would  not  make  the  corresponding  component 


of  r equal  to  zero  if  the  first  i components  of  h were  re- 
— n — n 


Specifically, 


placed  by  the  corresponding  components  of  hn+^ 
til 

the  i component  of  r^  computed  as  described  above  would  be 
less  than  zero  if  6 > 1 and  greater  than  zero  if  B <1.  There- 
fore, with  8 <1  the  algorithm  is  unciercorrecting  or  "under- 
relaxing" and  with  8 > 1 the  algorithm  is  overcorrecting  or 
"overrelaxing".  Convergence  of  the  algorithm  would  imply  that 
all  components  of  r would  approach  zero.  However,  due  to  the 
limitations  imposed  by  finite  precision  arithmetic,  r^  cannot 
be  made  exactly  equal  to  0^  in  practice  for  most  systems.  Thus, 
intuitively  it  can  be  seen  that  the  progress  that  the  SOR 
Method  makes  toward  a solution  at  each  step  is  directly  dependent 
on  the  parameter  8.  This  observation  is  confirmed  by  the  results 
concerning  the  rate  of  convergence  of  the  algorithm  presented 
in  Section  3.4.1. 

The  SOR  Method  is  another  example  of  a fixed-point  iter- 
ative algorithm  which  can  be  applied  to  the  adaptive  estimation 
problem.  It  has  been  shown  that  the  SOR  Method  converges  pro- 
vided that  the  relaxation  parameter  8 is  chosen  so  that 
0 < 8.  <2.  This  result  is  independent  of  the  statistics  of  the 
process  X(n).  In  addition,  a bound  on  the  rate  of  convergence 
of  the  algorithm  was  given.  A comparison  of  the  theoretical 
bounds  on  the  rates  of  convergence  of  the  mean-square  error  for 
the  SOR  Method,  the  Jacobi  Relaxation  Method  and  the  LMS  Grad- 
ient Method  will  be  presented  in  Chapter  V.  The  relationship 
between  the  theoretical  bounds  and  actual  performance  will  be 
studied  in  Chapter  VI  via  computer  simulation. 
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3.5  Number  of  Operations  Required  by  Adaptive  Fixed-Point 
Iteration 

As  mentioned  in  Chapter  II,  one  of  the  advantages  of  the 
LMS  Gradient  Algorithm  is  its  simplicity.  The  number  of  arith- 
metic operations  required  at  each  s iple  time  n is  on  the  order 
of  the  length  of  the  filter  N.  Obviously,  the  Adaptive  Fixed- 
Point  Iteration  Method  proposed  here  requires  a considerably 
larger  number  of  computations  at  each  sample  step.  In  this 
section,  the  number  of  operations  required  by  the  Adaptive 
Fixed-Point  Iteration  Method  will  be  determined  and  methods  for 
the  reduction  of  the  number  of  operations  will  be  explored. 

The  Adaptive  Fixed-Point  Iteration  Algorithm  can  be 
broken  down  into  three  procedures,  the  estimation  of  the  auto- 
correlation matrix  and  vector,  the  fixed-point  iteration,  and 
the  computation  of  the  filter  output.  As  mentioned  in  Section 


3.1,  the  elements  of  the  autocorrelation  matrix  and  the  auto- 
correlation vector  are  merely  sample  values  of  the  autocorrela- 
tion function  R^y(k)  for  stationary  processes.  Also,  since 


$xx  has  a Topelitz  structure  for  stationary  processes,  only  N 
samples  of  Rxx(k)  are  required  to  determine  all  of  the  elements 
of  the  autocorrelation  matrix.  In  addition  N samples  of  Rxx(k) 
are  needed  to  make  up  the  autocorrelation  vector  Rxx-  However, 


if  the  delay  n^  is  less  than  N,  then  only  (N+n^)  samples  of 
Rxx(k)  are  needed  to  determine  both  $xx  and  R^.  But,  if  n1>N, 
then  2N  samples  of  Rxx(k)  are  required  to  determine  both  the 


l 


autocorrelation  matrix  and  vector.  In  Section  3.1.3  it  was 
proposed  that  a moving  average  be  computed  recursively  to  obtain 
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an  estimate  of  the  autocorrelation  function.  From  (3.25)  it  cam 
be  seen  that  3 multiplications  and  2 additions  are  required  to 
update  the  estimate  of  R^fk)  for  each  required  value  of  k.  j 

Thus,  at  each  sample  time  n,  the  following  number  of  operations 
are  required  to  update  the  estimates  of  the  autocorrelation 
matrix  and  the  autocorrelation  vector: 


Delay  Value 

Additions 

Multiplications 

n!  _5  N 

2 (N+n^) 

3 (N+n^) 

n1  > N 

4N 

6N 

Figure  3.1  Required  number  of  operations/sample  for 

the  estimation  of  and  Rjcx  by  the  moving 
average  proposed  in  Section  3.1.3. 

Thus,  it  can  be  seen  from  Figure  3.1  that  the  number  of  opera- 
tions required  to  estimate  $xx  and  is  on  the  order  of  the 
length  of  the  filter  N for  the  moving  average  estimator  proposed 
in  Section  3.1.3. 

Estimation  of  the  autocorrelation  matrix  and  vector  re- 
quires on  the  order  of  N operations  per  sample  for  the  estimate 
considered  here.  Unfortunately,  the  fixed-point  iteration  pro- 
cedure requires  a considerably  larger  number  of  operations  per 
sample.  It  can  be  seen  from  (3.76)  and  (3.93)  that  if  these 
expressions  are  used  directly  to  implement  the  Jacobi  Relaxation 
Method  and  the  SOR  Method  respectively,  the  required  number  of 


computations  will  be  on  the  order  of  the  square  of  the  length 

2 

of  the  filter,  or  N , per  sample.  Figure  3.2  gives  the  specific 
number  of  operations  per  sample  required  for  the  Jacobi  Relaxa- 
tion Method  and  the  SOR  method  if  they  are  implemented  directly 
using  (3.76)  and  (3.93)  respectively. 


Algorithm 

Additions 

Multiplications 

Jacobi  Relaxation 

N (N+l ) 

N (N+l) 

SOR 

N (N+l) 

N (N+l ) 

Figure  3.2  Required  number  of  operations/sample  for 

Jacobi  Relaxation  and  SOR  Methods  computed 
directly  using  (3.76)  and  (3.93)  respective- 
ly. N = filter  length. 


The  final  procedure  of  the  Adaptive  Fixed-Point  Itera- 
tion method  is  the  computation  of  the  signal  estimate  or  the 
output  of  the  filter  by  the  use  of  equation  (1.4) . This  proce- 
dure requires  N multiplications  and  (N— 1 ) additions.  Thus,  the 
numbers  of  operations  required  for  Adaptive  Fixed-Point  Itera- 
tion using  the  Jacobi  Relaxation  Method  and  the  SOR  Method  are 
the  sums  of  the  numbers  of  operations  required  to  implement 
each  of  the  three  procedures  discussed  above.  Figure  3.3  gives 
the  total  required  number  of  operations/sample  for  Adaptive 
Fixed-Point  Iteration  using  the  Jacobi  Relaxation  Method  and  the 
SOR  Method. 
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Operation 

Jacobi  Relaxation 

SOR  Method 

rV_<  N 

nl  > N 

n.  j N 

n^  > N 

Additions 

B6 

N2+6N-1 

N2+4N+2n1-l 

N2+6N-1 

Multiplications 

N2+5N+3n1 

N2+8N 

N2+5N+3n1 

N2+8N 

Figure  3.3  Total  required  number  of  operations/sample  for 
the  Jacobi  Relaxation  Method  and  the  SOR  Method 
computed  directly  using  (3.76)  and  (3.93)  respect- 
ively with  estimation  of  the  autocorrelation 
matrix  and  vector  by  the  moving  average  proposed 
in  Sec.  3.2.1.  N = filter  length,  n.  = delay  in 
unit  sample  response. 

2 i 

From  Figure  (3.3)  it  can  be  seen  that  the  0 (N  ) operations  | 
per  sample  required  by  the  fixed-point  iteration  procedure 
dominates  the  expression  for  total  number  of  operations/sample 
for  both  the  Jacobi  Relaxation  and  the  SOR  Methods.  Thus,  a 
reduction  in  the  number  of  operations/sample  for  the  fixed-point 
iteration  methods  would  have  a significant  effect  on  the  amount 
of  computational  time  required  for  these  algorithms.  Methods 
for  the  reduction  of  the  number  of  operations  required  for  both 
the  iterative  methods  considered  here  are  discussed  in  the  fol- 

I 

lowing  sections . 

3.5.1  Reduction  of  Number  of  Operations/Sample  by  Use  of 

Partial  Iteration  Steps 

I 

As  stated  above,  both  the  Jacobi  Relaxation  Method  and 

2 

the  SOR  Method  require  0 (N  ) operations/sample  in  the  forms 
presented  in  (3.76)  and  (3.93).  Since  such  computational 
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complexity  may  be  prohibitive  in  many  applications,  it  is  of 
interest  to  consider  methods  for  reducing  the  number  of  opera- 
tions/sample. In  contrast  to  the  LMS  Gradient  Algorithm,  the 


"adaptation"  or  the  estimation  of  the  statistics  of  the  process 

X(n)  is  carried  out  independently  of  the  updating  of  the  unit- 

; sample  response.  Thus,  while  the  estimates  of  $ and  R are 

XX  XX 

updated  at  every  sample  time  n,  it  is  not  necessary  to  update 

all  of  the  elements  of  h^  by  the  fixed-point  iterative  procedure 

at  every  sample.  Therefore,  the  following  procedure  is  proposed 

for  updating  the  unit-sample  response  vector  h . After  k 

— n 

samples  have  been  received,  £ elements  of  will  be  updated 

and  the  process  repeated  every  k samples.  For  example,  with 

N=8,  k=5,  and  £=3,  after  5 samples  have  been  received  (n=4), 

the  first  3 components  of  h are  updated  (components  h^ , h2  and 

h^) • Then  after  5 more  samples  have  been  received  (n=9) , 

components  h4 , h^  and  hg  will  be  updated.  Finally,  at  n=14, 

the  next  three  components  (h?,  hg  and  h.^)  will  be  updated.  The 

process  continues  in  a similar  fashion  with  £=3  elements  of  h 

being  updated  in  order  every  k=5  samples.  When  the  end  of  the 

vector  is  reached,  the  first  element  becomes  the  next  in  order. 

That  is,  h^  follows  hN  in  being  updated.  If  this  scheme  is 

used,  then  the  average  number  of  operations  per  sample  for  the 

. 

fixed  point  iteration  portion  of  the  Adaptive  Fixed-Point 
Iteration  Method  is  on  the  order  of: 


l • (N/k) 


for  both  the  Jacobi  Relaxation  and  the  SOR  Methods.  The  specific 
average  number  of  operations/sample  under  this  scheme  is  given 
in  Figure  3.4  for  both  algorithms  considered  here. 


Operation 

Jacobi  Relaxation 

SOR  Method 

Additions 

Z/k  (N+l) 

Z/k  (N+l) 

Multiplications 

Vk  (N+l) 

Z/k  (N+l) 

Figure  3.4  Average  number  of  operations/sample  for 
Jacobi  Relaxation  and  SOR  Methods  with 
Partial  Iteration  steps.  N=filter  length, 
&/k=Partial  Iteration  Step  parameter. 


Obviously,  reduction  of  the  average  number  of  operations 
by  the  Partial -Iteration -Step  Method  will  slow  the  rate  of 
convergence  of  the  algorithms  proportionately  to  the  ratio 
Z/k).  Thus,  computational  complexity  and  rate  of  convergence 
can  be  traded  off  under  the  Partial-Iteration-Step  Method. 

The  measurement  of  computational  complexity  by  the  average  number 
of  operations/sample  can  be  justified  by  noting  that  the 
fixed-point  iteration  procedure  can  be  carried  out  by  spreading 
out  the  (£*N)  computations  over  the  next  k samples  since  the 
estimation  of  the  statistics  and  the  iteration  can  be  carried 
out  independently. 

Use  of  the  fixed-point  iteration  method  with  partial 
iteration  steps  is  one  approach  to  the  reduction  of  the  number 
of  arithmetic  operations  which  must  be  performed  at  each  sample 
time  n.  However,  as  mentioned  previously,  the  reduction  of 
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operations  by  the  use  of  partial  iteration  steps  reduces  the 
rate  of  convergence  of  the  algorithm.  Furthermore,  it  is 
interesting  to  note  that  while  the  Partial-Iteration-Step  Method 
discussed  above  was  suggested  as  a means  of  decreasing  the 
average  number  of  operations  per  sample,  the  same  procedure 
could  be  performed  with  £/k  > N so  that  the  number  of  operations 
per  sample  and  the  rate  of  convergence  is  increased.  The  limit- 
ing case  for  this  method  would,  of  course,  be  the  complete 
solution  of  the  system  of  linear  equations  at  every  sample  time 
n.  With  the  generalization  that  Z and  k can  take  on  any  integer 
values  greater  than  0,  the  method  discussed  above  will  be  re- 
ferred to  as  the  Variable  Iteration  Step  Method.  Thus,  Adap- 
tive Fixed-Point  Iteration  using  the  Variable  Iteration  Step 
Method  has  the  advantage  that  trade-offs  can  be  made  between 

I 

the  average  number  of  operations/sample  and  the  rate  of  conver- 
gence of  the  algorithm.  In  contrast,  the  LMS  Gradient  Algorithm 
requires  a fixed  number  of  operations  per  sample.  The  ability 
to  vary  the  number  of  operations/sample  in  the  Adaptive  Fixed- 
Point  Iteration  Method  is  a very  attractive  feature  since  for 
varying  applications  different  sampling  rates  will  be  used. 

The  average  number  of  operations/ sample  in  the  Variable  Iteration 
Step  Method  can  be  chosen  so  as  to  fully  utilize  the  time  availa- 
ble between  samples. 

The  theoretical  performance  of  Jacobi  Relaxation  and  the 

SOR  Method  under  the  Variable-Iteration-Step  scheme  will  be 

' 

studied  in  Chapter  V using  the  bounds  on  convergence  derived  in 
Section  3.2.3.  Specifically,  equation  (3.54)  can  be  used  to 
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generate  the  MSE  bound  recursively.  However,  the  sequence 

(h  »h*)  must  be  generated  by  the  particular  partial  iteration 
— n — 

method  as  discussed  above.  The  rates  of  convergence  for  various 
values  of  i and  k will  be  compared  to  the  rate  of  convergence 
of  the  LMS  Gradient  Algorithm.  Relationships  between  the 
theoretical  predictions  and  actual  performance  will  be  studied 
through  the  use  of  computer  simulation  in  Chapter  VI. 

3.5.2  Implementation  of  the  Jacobi  Relaxation  Method  Using  the 

Discrete  Fourier  Transform 

The  Variable-Iteration-Step  Method  discussed  in  the  last 
section  decreased  the  average  number  of  operations/sample  by 
choosing  1/k  < N and  thus  carrying  out  a single  iteration  step 
over  a number  of  samples  rather  than  making  a complete  iteration 
step  at  each  sample  time  n.  This  method  reduces  the  number  of 
operations/sample  at  the  expense  of  slowing  the  rate  of  con- 
vergence. Obviously,  it  would  be  advantageous  to  reduce  the 
number  of  operations  required  for  a complete  iteration  step. 

This  would  allow  more  complete  iterations  to  be  performed  under 
I the  Variable- Iteration- Step  Method  in  a given  number  of  sample 
periods  while  keeping  the  average  number  of  operations/sample 
constant.  In  this  section  it  will  be  shown  that  with  the 
estimate  of  the  autocorrelation  matrix  proposed  in  Section  3.1, 
the  Jacobi  Relaxation  Method  can  be  implemented  using  the 
Discrete  Fourier  Transform  (DFT) . Thus,  a Fast  Fourier  Trans- 
form (FFT)  algorithm  can  be  used  to  obtain  a substantial  reduc- 
tion in  the  number  of  operations  required  to  implement  Jacobi 


I 


I 


Relaxation.  This  result  arises  due  to  the  Topelitz  structure  of 
the  estimate  of  the  autocorrelation  matrix  and  does  not  hold  for 
the  general  case  of  a non-Topelitz  autocorrelation  matrix. 


The  alternate  form  for  the  Jacobi  Relaxation  Method  is 
given  in  equation  (3.76)  as: 

N _ 


fl  « yv  Q 

h.  . = h.  - ^-2—  Z 4.  . h.  + r. 

i,n+l  i,n  . , , 13  ,n  3 ,n  J i,n 

iin  J 


li  ,n 


where  h . , <(>..  and  R.  are  the  ifc^  component  of  the  unit- 
i i]  , n l ,n 

sample  response  vector  h,  the  (i,j)th  element  of  the  estimated 

^ th 

autocorrelation  matrix  $vv  and  the  i element  of  the  esti- 

aa  , n 

A 

mated  autocorrelation  vector  R^  n respectively.  Now,  from 
equations  (3.6)  and  (3.7): 


*ij  ,n  * 


Ri,n  RXX,n(nl+1_1) 


(3.6) 


(3.7) 


Thus,  substituting  for  „ and  R.  „ from  (3.6)  and  (3.7)  into 

i j / n i /H 

(3.76),  (3.76)  can  be  written  as: 


N 


h.  . . * h. 
i,n+l  i,n 


Rxx,n(0) 


1 h4  — ^vy  _ (i“  j ) 
j _ 2^  3 r n XX , n 


*XX,»(01 


RXX,n(nl+i“1) 


(3.94) 


Now,  recalling  from  equation  (1.5)  that  the  i component  of 
the  unit-sample  response  vector  is  the  i non-zero  sample  of 
the  unit-sample  response,  h^  n cam  be  written  as: 


h.=  h (n. +i-l) 

1 , n 1 n 


Thus,  equation  (3.76)  for  Jacobi  Relaxation  can  be  written  in 
terms  of  the  unit  saunple  response  h(i)  as: 


h(i+nl)n+l  = h(i+hl>n  - 


T 2 h(j+n,)  RxX  ,n(i-j) 

^Scx,n(0)  j=o 


7— Ryy  n (i+n,  )n  0 . < i < N - 1 

*«,n(0>S“'n 


(3.95 


Consideration  of  equation  (3.95)  reveals  the  basis  behind  the 
use  of  the  Discrete  Fourier  Transform  for  implementing  the 
Jacobi  Relaxation  Method.  The  summation  in  equation  (3.95): 


requires  N multiplications  and  (N-l)  additions  for  each  value 
of  0 < i _<  N-l  if  computed  directly.  However,  this  term  is 
merely  the  linear  convolution  of  two  sequences  h(i+n^)  and 
Rxx ( ^ ) n ‘ Thus,  it  can  be  computed  using  the  DFT.  However, 
(3.95)  is  not  in  the  simplest  form  for  computation.  Equation 
(3.95)  can  be  rewritten  as: 


h(1+nl)n+l 


■W0’ 


Rxx<i+nl>n  + ll-8)h(i+nl,„ 


[V  h(j+ni)n  jV1h'i+hi'n*xx.n(i-3)I 

(3.96) 


If  two  new  N-point  sequences  are  defined  as 


RL(i) 


R2(i) 


^X  #n 


-n 


(i)  0 . < i < N-l 


otherwise 


(i)  0 < i < N-l 


otherwise 


(3.97) 


(3.98) 


then  (3.91)  can  be  expressed  as: 


h(i+nl)n+l 


'Six  .n 


Rxx(i*n1)+d-B)h(i+nl)n 


pH (Qjt  Z h(3+ni)n  R.(i-j)  + Z h(j+n,)  R2(i-j)  ].  (3.99) 

AX,n  j=0  i=0 


Now,  hfi+n.^)^  R^(i)  and  R2(i)  are  N-point  sequences.  Therefore, 

the  linear  convolution  of  h(i+n..)  and  R. (i)  or  R_(i)  is  a 2N-1 

in  1 2 

point  sequence.  Thus,  (3.9  ) can  be  rewritten  as: 


h(i+nl>n+l 


^X  ,n 


1RXX,n(i+nl)  + ll-6)h(i+nl)n 


[h (i+n1 ) _*R,  (i)  + h(i+n.)*  R- (i) ] ] RECT„ (i)  (3.100) 

i n i in  £ n 


* 


where  the  * denotes  linear  convolution  and  the  sequence  RECT^j  ( i ) 
is  defined  as: 

1 0 _<  i _<  N-l 

RECTn ( i ) = 

0 otherwise 

Thus,  in  order  to  find  the  N-point  sequence  h(i+n^)n+^,  it  is 
necessary  to  compute  the  following  2N-1  point  sequence: 

y (i)  n = [hd+n^  *R1  (ij+hd+n^  *R2  (i)  ] 

and  truncate  the  sequence  after  the  first  N points.  Thus,  if 
the  DFT  is  used  to  compute  y(i)n,  then  a DFT  of  at  least  2N-1 
points  will  be  required  to  represent  the  linear  convolution  of 
the  sequences.  Suppose  that  Discrete  Fourier  Transforms  of 
p >_  2N-1  points  are  computed  for  the  N-point  sequences  h(i+n^), 
R1(i)  and  R2(i)  where  the  remaining  (p-N)  points  are  chosen  to 
be  zero.  The  p-point  DFT  A(k)  of  an  N.  <p  point  sequence  A(i) 
padded  with  (p-N)  zeroes  is  defined  as: 

N-l  -illki 

A(k)  =»  E A (i ) e J p * k-0,1,...  ,p-l  . (3.101) 

i*0 

Provided  that  the  DFTs  of  the  N-point  sequence  h(i+n1>n,  R^n) 

and  R2  (n)  are  computed  as  described  above,  the  p 2N-1  point 

DFT  of  the  2N-1  point  sequence  Ynd)  is  given  by  the  product  of 

the  DFTs  of  h(i+n. ) and  (R. (i) +R0 (i) ) . Thus,  since  the  DFT  is 

in  l 2 

a linear  operator: 


(3.102) 


Y (k)  - H(k)  (R.  (k)+R,(k)  ) 

n n l i. 

Now,  it  can  be  shown  that  R2(k)=R*(k)  where  * denotes  the  complex 
conjugate.  Thus,  (3.102)  can  be  rewritten  as: 


Y(k)  = H(k)  (R.  (k)  +R*  (k) ) 
n n 1 1 

or  equivalently: 

Y (k) _ = 2H (k)  Re [R.  (k) ] (3.103) 

n n 1 

where  Re[R1(k)]  is  the  real  part  of  R1 (k) . Thus,  the  following 
procedure  is  proposed  for  computing  h^+^  by  the  Jacobi  Relaxation 
Method  assuming  that  R^ (k)  is  obtained  from  the  autocorrelation 
estimate  proposed  in  Section  3.1: 


Step  1: 

Step  2 : 
Step  3: 

Step  4 : 


Compute  the  p >_  2N-1  point  DFT  of  the  N-point 

sequence  h(i+n.)  to  find  H(k)  . 

in  n 

Compute  Y(k)n  by  (3.103). 

Take  the  inverse  DFT  of  Y(k)  to  obtain  the 

n 

2N-1  point  sequence  y(i)n» 

Compute  the  N-point  sequence  h(i+n. ) . as: 


h(l+nl)n+l 


[RXX(l+nl)n"y (l)n] 


RXX(0)n 

The  reduction  of  the  number  of  operations  required  to 


(3.104) 


+ ( 1-6  ) h (i+n.  ) 
i n 


carry  out  one  step  of  the  Jacobi  Relaxation  Method  comes  when 
the  DFT  computations  in  the  above  method  are  made  using  an  FFT 
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algorithm  used  for  the  computation.  Usually,  however,  the 
number  of  additions  and  multiplications  are  on  the  order  of 
p log(p).  In  the  discussion  that  follows,  it  will  be  assumed 
that  N is  an  integer  power  of  2 and  that  a 2N  point  DFT  is  to 
be  used  in  the  computations  required  to  implement  the  Jacobi 

Relaxation  Method  as  proposed  above.  It  is  further  assumed  that  I 

| 

a decimation-in-time  (DIT)  FFT  algorithm  with  in-place  computa- 
tions such  as  that  described  in  [21]  is  used  to  compute  the  DFT. 
For  a p-point  DFT  computed  using  the  DIT  FFT  algorithm  consi- 
dered, p log2(p)  complex  additions  and  p/2  log2 (p)  complex 
multiplications  are  required.  Thus,  to  compute  a 2N-point 
DFT  would  require  2N  (l+log2(N))  complex  additions  and  n 
(l+log2(N))  complex  multiplications.  However,  for  Step  1 of  the 
procedure,  the  2N-point  DFT  is  taken  for  an  N-point  sequence 
padded  with  N zeroes.  Thus,  the  number  of  operations  required 
for  Step  1 is  less  than  that  required  for  a general  2N-point 
DFT.  Specifically,  2N  log2 (N)  complex  additions  and  N log2 (N) 
complex  multiplications  are  required  to  carry  out  Step  1. 

Step  2 is  simply  the  multiplication  of  2 2N-point  trans- 
forms and  thus  requires  2N  complex  multiplications  and  2N  com- 
plex additions.  The  complex  additions  are  used  in  lieu  of  a 
multiplication  by  the  real  constant  2. 

Step  3 requires  the  computation  of  an  inverse  DFT  which 
can  also  be  carried  out  using  the  same  2N-point  FFT  algorithm 
discussed  above.  Thus,  2N  (l+log2(N))  complex  additions  and 
N (l+log2(N))  complex  multiplications  are  required.  However,  the 
number  of  additions  can  be  reduced  by  noting  that  only  the  first 
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N-points  of  the  2N-point  result  will  be  used  in  Step  4.  Thus, 
the  number  of  complex  additions  can  be  reduced  to  N (l+log2(N)). 

Finally,  Step  4 is  simply  the  addition  of  three  N-point 
sequences  and  the  intermediate  multiplications  by  two  real  con- 
straints. Thus,  2 (N-l)  real  additions  and  2N  real  multiplications^ 
are  required  to  complete  Step  4. 

The  total  number  of  operations  required  to  implement  the 
Jacobi  Relaxation  Method  using  the  procedure  proposed  is  the 
sum  of  the  numbers  of  operations  required  for  Steps  1,  2,  3, 
and  4.  Since  the  other  algorithms  discussed  here  require  only 
real  operations,  the  number  of  complex  operations  required  for 

i 

the  method  proposed  will  be  converted  into  the  equivalent  number 
of  real  operations.  Since  two  real  additions  are  required  for 
each  complex  addition  and  four  real  multiplications  and  two  real 

I 

additions  are  required  for  each  complex  multiplication,  the 
number  of  operations  given  in  the  above  discussion  must  be 
increased  accordingly.  Figure  3.5  gives  the  number  of  real 
arithmetic  operations  required  to  implement  each  step  of  the 
proposed  procedure. 


Operation 

Step  1 

Step  2 

Step  3 

Step  4 

Additions 

6N  Log2N 

8N 

6N(l+Log2N) 

2 (N-l ) 

Multiplications 

4N  Log2N 

8N 

4N(l+Log2N) 

2 N 

Total 


16N+12N  Log_N- 2 

^ \\ 


14N+8N  Log2N 


Figure  3.5  Number  of  real  operations  required  to 

implement  Jacobi  Relaxation  using  the  FFT . 
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From  Figures  3.2  and  3.5  it  can  be  seen  that  for  small 
values  of  the  filter  length  N it  is  advantageous  to  do  the 
required  computations  for  Jacobi  Relaxation  directly  using  (3.76) 
Nowever,  as  N becomes  larger  (N  >_  64),  substantial  savings  in 
the  number  of  operations  required  can  be  obtained  by  the  use  of 
the  FFT  as  outlined  above.  For  example,  with  N=1024,  approx- 
imately 1.048*106  real  multiplications  are  required  to  carry  out 

one  step  of  Jacobi  Relaxation  directly  using  (3.76).  In  con- 

4 . 

trast,  roughly  9.523*10  real  multiplications  are  required  when 

the  FFT  is  employed.  Thus,  when  impelmented  using  the  FFT,  the 

Jacobi  Relaxation  Method  requires  a number  of  arithmetic  compu- 

2 

tations  which  is  between  N and  N . 

In  the  above  discussion  it  was  assumed  that  the  DFT  of 
the  sequence  R^(i)  defined  in  (3.97)  was  known.  It  will  now  be 
shown  that  Ri (k)  can  be  determined  recursively  without  having 
to  perform  an  additional  DFT  at  each  step.  From  (3.23)  and 
(3.97)  it  cam  be  seen  that: 

0 < i _<  N-l 

otherwise. 


R,  (i)  3 rr  Z X(H)X(Jl-i) 
1 M 4-n-M+l 

0 


Thus,  from  the  definition  of  the  DFT  (3.101),  the  p-point  DFT 
of  R^(i)n  padded  with  (p-N)  zeroes  can  be  written  as: 


N-l  . n 

R,  (k)  = E [i  I 

1 n i-1  M Jl=n-M+1 


X(4)X(4-i) ]e 


j— 

P 


(3.105) 
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By  interchanging  the  order  of  summation,  (3.105)  can  be  rewritten 


. n N-l  -j—  ki 

R,  Oc)  “ S 1 XU)  s XU-i)e  P 
1 n M A=n_M+1  i=1 


or,  equivalently. 


Vk)„ 


n N-l  -j— ki 

s X(M  [ Z XU-i)e  P -XU)  ] 


(3.106) 


&=n-M+l 


Now,  define  a new  sequence  as: 


..  1 . 2 TT  . 

N-l  -3—  kx 

A?  (k)  = Z XU-i)e  P -XU) 

16  i=0 


(3.107) 


so  that  R^(k)n  may  be  expressed  as: 


R.  (k)  - ± Z XU)A.  (k) 

1 n M *=n-M+l  * 


(3.108) 


Thus,  from  (3.108)  it  can  be  seen  that  R^(k)n+i  can  be  written 


Rl(k)n+1  = Rl(k)n  + g tX(n+l)An+1(k)-X(n-M+l)An_M+1(k) ] . (3.109) 


n /1.N  aUi.  . j j ; r « /i-  \ — „ i 


By  introducing  the  change  of  variables  4=i-l,  the  above  expres- 
sion can  be  rewritten  as: 

-j^k  N-l  -j^k£  “j^jkN 

A (k)  - e p Z X (n-X.)  e p -e  p X(n-N+1) 
n+1  £=0 


or  equivalently: 


Vi (k)  = e 


N-l  -j%c*  -3%N 

p Z X(n-£)e  p -e  p X(n-N+1) 


From  the  above  expression  it  can  be  seen  that  An+1(k) 
can  be  expressed  recursively  as: 

-j^k 

An+i(k)  » e p (An(k)+X(n) )-e  p (X(n-N+1)).  (3.110) 


Thus,  R^(k)n  can  be  updated  recursively  requiring  on  the  order 
of  p operations/s ample  rather  than  the  0 (p  log(p))  operations 
required  to  compute  R^(k)  directly  using  an  FFT  algorithm.  In 
the  special  case  considered  previously  where  p=2N,  (3.110)  may 
be  reduced  to  the  following  form: 

-jjj*  k 

A^+1(k)  -e  N (An(k)+X(n) )-(-!) * X(n-N+1) . (3.111) 


Therefore,  updating  An(k)  requires  4N  real  additions  and  2N 

complex  multiplications  at  each  sample  step. 

By  considering  (3.109)  and  the  above  discussion,  it  can  be 

seen  that  updating  Re [R. (k)  ] requires  12N  real  multiplications 

n 

and  8N  real  additions  which  is  considerably  less  than  the 


c 


N log2(N)  complex  operations  required  to  compute  the  DFT  using  a 
FFT  algorithm  when  N > 8. 

The  computations  required  by  the  Jacobi  Relaxation  Method 
can  be  carried  out  using  the  FFT  method  described  in  this 
section  with  recursive  updating  of  RetR^UO^  to  obtain  a sub- 
stantial reduction  in  the  number  of  operations/iteration  step 
over  the  direct  computation  of  (3.76)  for  N >_  64.  For  large  N 
the  number  of  operations  required  is  dominated  by  the  number  of 
computations  required  for  the  FFT  algorithm  which  is  on  the 
order  of  N log(N).  Thus,  Jacobi  Relaxation  realized  by  this 
method  lies  between  the  LMS  Gradient  Algorithm  and  the  SOR 
method  in  terms  of  the  number  of  operations  required  per  itera- 
tion step.  To  further  reduce  the  average  number  of  operations/ 
sample,  the  FFT  method  can  be  combined  with  the  Variable  Itera- 
tion Step  Method  proposed  in  Section  3.5.1.  Since  the  majority 
of  the  operations  required  by  the  FFT  method  are  executed  in  the 
computation  of  the  FFT  it  is  most  useful  to  update  all  N terms 
of  the  unit  sample  response  vector  every  k samples.  That  is 
&=N  in  the  Variable  Iteration  Step  Method.  In  this  case,  the 
approximately  N log2 (N)  operations  are  distributed  over  k samples 
for  an  average  of  N/k  log2(N)  operations  per  sample.  Thus,  the 
FFT  method  for  Jacobi  Relaxation  can  be  used  to  reduce  the  re- 
quired number  of  operations/sample  either  by  itself  or  in  con- 
junction with  the  Variable-Iteration-Step  Method  obtaining  a 


substantial  savings  in  computation  time  for  large  N provided  that 
the  autocorrelation  estimate  yields  an  autocorrelation  matrix 
with  a Topelitz  structure. 
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3.5.3  Reduction  of  Computation  Time  and  Parallel  Processing 

The  FFT  Method  for  Jacobi  Relaxation  and  the  Variable 
Iteration  Step  Method  for  the  SOR  Method  and  Jacobi  Relaxation 
are  aimed  at  the  reduction  of  the  number  of  operations  re- 
quired per  sample  as  a means  for  reducing  the  amount  of  computa- 
tional time  required  by  the  algorithms.  Another  approach  to  the 
reduction  of  computation  time  is  to  increase  the  speed  with 
which  the  required  group  of  operations  can  be  carried  out.  One 
method  for  achieving  this  goal  is  to  carry  out  several  operations 
concurrently  or,  to  process  in  parallel.  Adaptive  Fixed-Point 
Iteration  is  well  suited  to  parallel  processing  since  the  re- 
quired tasks  can  be  divided  into  three  distinct  areas,  the  up- 

A A 

dating  of  the  estimates  $vv  and  R™.  , the  fixed-point  itera- 

tion,  and  the  computation  of  the  filter  output.  In  addition, 
the  fixed-point  iteration  algorithms  discussed  here  are  also 
well  suited  to  parallel  processing  in  themselves  [22].  Thus, 


while  not  of  primary  interest  here,  parallel  processing  is  a 
viable  method  for  the  reduction  of  computational  time  in  Adaptive 
Fixed-Point  Iteration. 


Another  approach  for  the  reduction  of  computational  time 
cam  be  developed  by  noting  that  the  majority  of  the  computations 
required  by  the  Jacobi  Relaxation  Method  and  the  SOR  Method  are 
involved  with  computing  sums  of  the  form: 


*i 


N-l 

Z 

i-0 


<t> . . 

ID 


which,  in  terms  of  the  unit  sample  response  and  the  autocorrela- 
tion function  estimate  proposed  in  Section  3.1,  can  be  written 
as: 

n2 

y (j)  = 2 

i=n 

Now,  the  above  expression  is  identical  to  the  convolution  sum 
required  to  produce  the  filter  output  except  that  X(i-j)  is 
replaced  by  R^  n(j-i+n1).  Thus,  the  filter  structure  itself  can 
be  used  to  carry  out  part  of  the  computations  required  to  up- 
date the  unit  sample  response.  Since  in  many  cases  high  speed 
special  purpose  hardware  is  used  to  implement  the  transversal 
filter,  this  approach  may  allow  very  rapid  computation  of  the 
sums  required. 

Thus,  parallel  processing  and  use  of  any  special  hardware 
used  for  implementing  the  filter  are  two  approaches  to  the 

I 

reduction  of  the  time  required  to  carry  out  the  necessary 
computations  for  Adaptive  Fixed-Point  Iteration  using  the  Jacobi 
Relaxation  Method  and  the  SOR  Method.  Again,  as  was  the  case 
with  the  FFT  method,  either  of  these  two  approaches  can  be  used 

I 

in  conjunction  with  the  Variable  Iteration  Step  Method  to  reduce 
the  average  number  of  computations  per  sample  step. 

3.6  Conclusion 

Adaptive  Fixed-Point  Iteration  has  been  proposed  as  a 


h (i) Rxx  n ( j“i+ni)  • (3.112) 


method  for  the  approximate  solution  of  the  Wiener-Hopf  equation 
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for  a finite  impulse  response  digital  filter.  A direct  estimate 

which  is  consistent  was  chosen  for  the  estimation  of  the  auto- 

■ 

correlation  matrix  $xx  and  the  autocorrelation  vector  for 
the  case  of  a stationary  input  process  X(n).  For  the  more  general 
case,  a moving  average  was  proposed  as  an  estimate  of  these 
statistics  and  a bound  on  the  variance  of  this  estimate  as  a 
function  of  the  length  M of  the  data  window  was  derived.  In 
this  development  it  was  assumed  that  the  statistics  of  the  pro- 
cess X(n)  vary  slowly  when  compared  to  M and  to  the  filter  length 
N.  This  estimate  yields  an  estimated  autocorrelation  matrix 
having  a Topelitz  structure  which  is  advantageous  in  processing. 
However,  the  Adaptive  Fixed-Point  Iteration  approach  is  also 
applicable  when  other  estimates  which  do  not  force  such  a struc- 
ture on  the  autocorrelation  matrix  are  used. 

A general  form  for  fixed-point  iteration  algorithms  was 
presented  and  conditions  for  convergence  in  the  mean  were 
established.  In  addition  a bound  on  the  rate  of  convergence 

i 

of  the  mean  square  error  to  the  minimum  MSE  produced  by  the  opti-  j 
mal  filter  was  established.  Two  particular  examples  of  fixed- 
point  iteration  were  considered  in  detail.  Specifically,  the 
Jacobi  Relaxation  Method  and  the  SOR  Method  were  proposed  as 
algorithms  for  the  implementation  of  an  Adaptive  Fixed-Point 
Iteration  scheme.  Conditions  on  a parameter  B present  in  both 

I 

algorithms  were  determined  to  insure  convergence;  independent 
of  the  statistics  of  the  process.  Two  schemes  for  the  reduction 
of  the  average  number  of  operations  per  sample  were  presented. 


The  Variable  Iteration  Step  Method  is  applicable  to  both  algor- 
ithms while  the  FFT  Method  applies  only  to  Jacobi  Relaxation 
used  in  conjunction  with  an  autocorrelation  matrix  estimate 
whicn  has  a Topelitz  structure.  Also,  the  use  of  parallel  pro- 
cessing and  any  available  special  purpose  transversal  filter 
hardware  for  the  reduction  of  the  computation  time  required  was 
discussed  briefly. 

It  is  believed  that  Adaptive  Fixed-Point  Iteration  offers 
an  extremely  attractive  solution  to  the  problem  of  adaptively 
determining  an  approximation  to  the  optimal  filter  when  some 
increase  in  complexity  over  the  LMS  Gradient  Algorithm  can  be 
tolerated.  In  particular,  the  Jacobi  Relaxation  Method  and  the 
SOR  Method  can  be  made  to  converge  by  a choice  of  the  relaxation 
parameter  6 which  is  independent  of  the  statistics  of  the  input 
process  X(n).  In  addition,  the  use  of  the  Variable  Iteration 
Step  Method  makes  Adaptive  Fixed-Point  Iteration  extremely  flex- 
ible in  that  trade-offs  can  be  made  between  the  rate  of  con- 
vergence and  the  average  number  of  operations  required  per  sam- 
ple. Comparisons  between  the  theoretical  performance  of  the  LMS 
Gradient  Algorithm,  the  Jacobi  Relaxation  Method  and  the  SOR 
Method  will  be  considered  in  Chapter  V.  Relationships  between 
the  theoretical  and  actual  performance  of  the  three  algorithms 
will  be  explored  via  computer  simulation  in  Chapter  VI.  In  all 
cases,  rate  of  convergence  and  required  number  of  operations 
per  sample  will  be  the  variables  of  interest.  It  will  be  shown 
in  these  chapters  that  Adaptive  Fixed-Point  Iteration  offers 


CHAPTER  IV 

NON-ITERATIVE  SOLUTION  — THE  LEVINSON  ALGORITHM 

The  LMS  Gradient  Algorithm  discussed  in  Chapter  II  and 
the  Adaptive  Fixed-Point  Iteration  Method  proposed  in  Chapter 
III  take  somewhat  similar  approaches  to  the  approximate  solution 
of  the  discrete  finite  impulse  response  Wiener-Hopf  equation 
given  by  (1.13).  These  methods  are  similar  in  that  both  make 
estimates  of  the  statistics  of  the  process  and  attempt  to  find 
an  approximate  solution  to  the  problem  by  the  use  of  an  iterative  j 
technique.  The  LMS  Gradient  Method  uses  an  instantaneous  esti- 
mate of  the  gradient  of  the  mean-square  error  and  applies  a 
Steepest  Descent  type  method  at  each  sample  point  while  the 
Adaptive  Fixed-Point  Iteration  Algorithm  uses  estimates  of  the 
autocorrelation  matrix  and  vector  and  attempts  to  solve  a system 
of  the  form: 

*xx*  = *xx  (3*26) 

by  applying  a complete  or  partial  step  of  a fixed-point  iteration 
method  every  k sample  periods.  The  estimates  of  the  autocorrela- 
tion matrix  and  the  autocorrelation  vector  used  in  (3.26)  are 
restricted  only  in  that  they  should  be  at  least  asymptotically 


L29 


* 


130  | 

unbiased  and  have  a small  covariance.  No  special  structure  is 

imposed  upon  the  estimate  of  the  autocorrelation  matrix  in  order 

* 

for  the  Adaptive  Fixed-Point  Iteration  Method  to  be  applicable. 

One  possible  estimate  for  use  in  Adaptive  Fixed-Point  Iteration 
was  proposed  in  Section  3.1.  This  particular  estimate  imposes 
a Topelitz  structure  on  *xx-  That  is,  each  element  of  $xx  can 
be  defined  as : 

- f(i-j)  (4.1) 

where  f is  some  function  of  (i-j).  Specifically,  for  the  esti- 

I 

mate  considered  in  Section  3.1,  <)k  ^ is  given  by  (3.6)  as: 

♦ij  ' • (3-6) 

The  Topelitz  structure  of  the  autocorrelation  matrix 
estimate  given  by  (3.6)  led  to  a reduction  of  the  number  of 
operations  required  to  implement  each  step  of  the  Jacobi  Relaxa- 
tion Method  as  discussed  in  Section  3.5.2.  However,  the  Tope- 
litz structure  also  makes  possible  the  use  of  very  efficient 
non-iterative  algorithms  for  the  solution  of  the  system  of 

equations  given  by  (3.26).  In  this  chapter,  the  Levinson  Al- 

/\ 

gorithm  for  the  solution  of  the  system  (3.26)  where  $xx  is  a 
Topelitz  matrix  will  be  considered  [23].  This  algorithm  is 
non-iterative  in  that  the  solution  of  (3.26)  is  obtained  in  a 
fixed  number  of  steps.  As  will  be  shown  later,  the  Levinson 
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Algorithm  requires  on  the  order  of  N operations  to  completely 
solve  (3.26)  where  $xx  is  a Hermetian  Topelitz  matrix. 


4.1  The  Levinson  Algorithm 

The  exact  solution  of  a system  of  N linear  equations  of 
the  form  (3.26)  requires  on  the  order  of  N3  operations  in  the 
general  case  when  using  any  of  the  standard  methods  for  solving 
linear  equations.  Iterative  methods  require  fewer  operations 
if  the  convergence  rate  is  high  enough  but  do  not  yield  exact 
solutions.  However,  if  the  system  in  (3.26)  is  Hermetian  and 
Topelitz,  then  the  solution  can  be  found  most  efficiently  by 
the  use  of  Levinson's  Algorithm  [23].  In  the  non-Hermitian  case, 
the  Trench  Algorithm  can  be  used  [24].  These  algorithms  util- 
ize the  fact  that  the  first  row  and  column  of  the  inverse  of  a 
Topelitz  matrix  uniquely  determine  the  remaining  elements  of  the 
inverse.  The  use  of  the  Levinson  or  Trench  Algorithms  leads  to 
substantial  savings  in  the  number  of  operations  required  for  the 
exact  solution  of  (3.26). 

Since  the  autocorrelation  matrix  estimate  proposed  in 

A * 

Section  3.1  produces  a matrix  $xx  which  is  Hermitian  and  Tope- 
litz, the  Levinson  Algorithm  can  be  considered  as  a method  for 
the  solution  of  (3.26)  when  this  estimate  is  used.  The  Levinson 
Algorithm  is  recursive  in  nature.  However,  the  exact  solution 
of  (3.26)  is  obtained  in  a fixed  number  of  steps.  In  the  dis- 

A A 

cussion  that  follows,  it  will  be  assumed  that  $xx  and  Rxx  are 
fixed  during  the  application  of  the  Levinson  Agorithm  and  that 
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the  following  definitions  apply: 


= 

T 

[hi '^2 ' * * * '^i^ 

(4.2) 

a . = 

— i 

[RjQjd),  Hxx(2)f...#Rxx(i)]T  • — — 

1^(0) 

(4.3) 

k = 

[^xx (i) ' Rxx(i-l>r-..#Rxx(l)]T  • ~ — - — 

Rxx{0) 

(4.4) 

n 

•H 

trl 

T 

[g^'9 2' ^3' * * * 

(4.5) 

✓ 

il 

T 

' * * * ' ^1  ^ 

(4.6) 

As  in  any  recursive  algorithm  initial  values  are  assigned  in 
the  Levinson  Algorithm  before  the  recursion  begins.  These 
initial  values  are  assigned  as  follows: 

-1  * RXx(ni)//®xx(0)  (4,7) 

k = -*xx(1)/iScx(0)  * -ai  • (4-8) 

With  the  above  definitions  and  initial  values,  the  recursion 
relationships  for  Levinson's  Algorithm  are  as  follows: 


Gi  - Rxx(ni+1)/Rxx(0) 
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hi=  -Rxx(i+1)/Rxx(0)  - 


T ~ 

hi  SLi 


X.  = 1 + a.  g. 
1 —l 


— i+1 


h . + 0 . /X . g . 

— l l l 


ai+i 


0./X. 


n • /X . 

i i 


a+  n . /x . g . 

i i i 2-1 


(4.9) 

(4.10) 

I 

(4.11) 

(4.12) 


(4.13) 


When  the  recursive  relationships  in  (4.9)  through  (4.12) 
are  carried  out  for  i = 1,2 , 3, . . . ,N-1  the  final  result  is 
the  solution  to  the  system  (3.26). 

4.2.1  Proof  of  Levinson's  Algorithm 

The  fact  that  h^  produced  by  Levinson's  Algorithm  is 
the  solution  to  the  N order  system  (3.26)  is  not  obvious  from 
an  inspection  of  the  recursive  relationships  in  (4.9)  through 
(4.12).  Therefore,  the  following  inductive  proof  is  offered  as 
a verification  of  the  algorithm.  It  will  be  assumed  without 
loss  of  generality  that  the  system  of  (3.26)  has  been  normalized 
by  dividing  both  sides  by  Rxx(0)  so  that  the  resulting  normal- 
ized system  is: 


where  the  subscript  N indicates  that  the  system  is  of  N order 
and  the  following  definitions  apply: 


$ = 
N 


V0) 


XX 


(4.15) 


—XX 


(4.16) 
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Now,  consider  the  (N+l)  order  normalized  system  given  by 

$N+1  ^N+l  = ^n+1  * 

This  system  cam  be  partitioned  as  follows  due  to  the  Topelitz 


(4.17) 


structure  of  $N+1  and  $N: 


$N 

Sn 

^N+l 

^4 

H* 

1 

_fiXX<VN,/*XX(°i 

(4.18) 


Substitution  for  in  (4.18)  from  (4.12)  and  expansion  gives 

the  following  two  equations: 


$N  + 9n/XN  $N  + 0N/XN 


(4.19) 


S h„  + VXN  5^  3n  + VXN  * «xx'VN)/Sxx(0)-  (4'20> 


From  the  expressions  for  0N  and  XN  in  equations  (4.10)  and  (4.11) 


respectively,  it  can  be  seen  that  (4.20)  is  satisfied.  Thus, 
hN+^  is  the  solution  to  (4.17)  provided  that  (4.19)  is  satis- 
fied. Suppose  that  is  a solution  to  (4.14).  Then  (4.19) 
reduces  to: 


*N  % + 2n  * - 


(4.21) 


Therefore,  h^+^  is  a solution  to  (4.17)  provided  that  is  a 
solution  to  (4.14)  and  (4.21)  is  satisfied.  To  see  that  (4.21) 
is  satisfied  for  all  N,  consider  the  following  inductive  proof. 
Equation  (4.21)  can  be  partitioned  as: 


T 

1 2n-i 

r 

^N-l  $N-1 

2n  + 

^N-l 

= 0 


(4.22) 


Now,  substituting  for  in  (4*22)  from  (4.13)  and  expanding 
yields  the  following  two  expressions: 

nN-l/XN-l  + ^N-l  3^-1  + V-l^N-l  — N— 1 %- 1 + aN  = 0 (4.23) 

^N-l  tlN-l/XN-l‘+  ®N-1  Sn-1  + nN-l/XN-l  ®N-1  2n-1  + ^N-l  = 

(4.24) 

It  can  be  seen  from  the  expressions  for  nN-1  and  in  (4.10) 

and  (4.11)  respectively  that  (4.23)  is  satisfied.  Thus,  (4.21) 
will  be  satisfied  provided  that  (4.24)  holds.  Equation  (4.24) 
can  be  rewritten  as: 


N-l  2n-1  + 4l-l  + nN-l/XN-l  ($N-1  2n-1  + ^N-l}  - 


(4.2 


Now,  define  the  vector  b„  . as 

-N-l 


— N-l  $N-1  %-l  + ^N-l 


(4.2 


Due  to  the  fact  that  $N_^  is  symmetric  and  Topelitz,  b^^  can 
be  written  as: 


-N-l  $N-1  %-l  + ^N-l 


Thus,  (4.25)  can  be  expressed  as: 


-N-l  + nN-l'/XN-l  -N-l  - * 


Thus,  if  it  is  assumed  that  (4.21)  holds  with  N replaced  by 
N-l,  that  is  if: 


£n-i  = V-l  %-!  + In-1  - » 

then  (4.25)  will  be  satisfied.  Thus,  since  it  has  been  shown 
that  (4.21)  holds  for  all  N provided  that  it  is  true  for  N-l, 
by  induction  (4.21)  will  be  satisfied  for  all  N provided  that 
it  is  satisfied  for  the  case  N=l.  That  is  if: 


It  can  be  seen  from  the  initial  value  assigned  in  (4.8)  to  ^ 
that  (4.27)  holds.  Thus,  by  induction  (4.21)  holds  for  all  N. 


It  has  been  shown  that  h.TJ,  is  the  solution  to  (4.17) 
provided  that  is  a solution  to  (4.14).  Thus,  by  induction 
hjj  is  a solution  to  (4.14)  for  all  N provided  that  it  is  the 
solution  in  the  case  where  N=l.  That  is  if: 


$i  £i  = *i 


(4.28) 


Since  the  system  (4.14)  was  assumed  to  be  normalized,  (4.28) 
reduces  to : 


It  can  be  seen  from  the  initial  value  assigned  in  (4.7)  and  the 
definition  in  (4.16)  that  (4.28)  holds.  Thus,  it  has  been  proved 
by  induction  that  h^  is  the  solution  to  (4.14)  and  therefore 
to  (3.26). 

While  not  obvious  from  the  recursive  relationships  in 

(4.9)  through  (4.13),  h^j  produced  by  Levinson's  Algorithm  is 

'fcll 

indeed  the  solution  to  the  N order  linear  system  of  equations 
in  (3.26)  provided  that  $xx  is  constrained  to  be  Hermitian  and 
Topelitz  as  shown  in  the  above  proof.  The  fact  that  this  al- 
gorithm can  be  used  to  solve  the  system  (3.26)  leads  to  sub- 
stantial savings  in  the  number  of  computations  over  other  non- 
iterative methods  as  will  be  seen  in  the  next  section. 
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4.1.2  Number  of  Operations  Required  by  Levinson's  Algorithm 

The  solution  of  the  Nth  order  system  of  linear  equations 
(3.26)  by  a standard  method  such  as  Gaussian  Elimination  requires 
on  the  order  of  N2  operations.  Such  computational  effort  be- 
comes unreasonable  for  even  moderately  large  values  on  N.  How- 

A 

ever,  in  the  special  case  where  4>xx  is  Hermitian  and  Topelitz, 

the  use  of  Levinson's  Algorithm  can  substantially  reduce  the 

amount  of  computational  effort  required. 

Inspection  of  the  recursive  relationships  (4.9)  through 

(4.13)  and  the  initial  values  (4.7)  and  (4.8)  reveal  that  on  the 
2 

order  of  N operations  are  required  for  the  solution  of  the 
Hermitian  Topelitz  system  (3.26)  using  Levinson's  Algorithm. 

The  specific  number  of  real  arithmetic  operations  required  is  . * 
given  in  Figure  4.1. 

The  number  of  operations  required  by  Levinson's  Algorithm 
. . 2 

is  dominated  by  the  term  2 . 5N  for  large  N as  can  be  seen  in 
Figure  4.1.  The  required  number  of  operations  can  be  reduced 
somewhat  by  the  use  of  a modification  of  Levinson's  Algorithm 


Additions 

Multiplications 

Divisions 

2.5N2+0.5N-3 

2.5N2-2.5N 

4N-2 

Figure  4.1  Number  of  Operations  Required  for  the 

Solution  of  a Hermitian  Topelitz  System 
by  Levinson's  Algorithm. 
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due  to  Zohar  [25].  This  modification  replaces  equation  (4.11) 
for  the  computation  of  the  parameter  X^  with  the  recursive 
relationship : 


X.  . . = X.  - nf/X. 

i+l  l i'  l 


(4.29) 


where  X is  initialized  at  the  start  of  the  algorithm  to  be: 

\1  * 1 - (Rxx(1)/^XX(0))2  = 1 " al  ' (4.30) 

With  this  modification,  the  solution  of  the  system  (3.26) 

2 

requires  approximately  2N  operations  for  large  N.  The  exact 
number  of  operations  required  by  the  Modified  Levinson's 
Algorithm  is  given  in  Figure  4.2. 


Additions 

Multiplications 

Divisions 

2N2+N-2 

2N2+2N-4 

4N-2 

Figure  4.2  Number  of  Operations  Required  for 

the  Solution  of  a Hermitian  Topelitz 
System  by  Levinson's  Algorithm  with 
Modification  due  to  Zohar. 

From  Figures  4.1  and  4.2  it  can  be  seen  that  the  use  of 
Levinson's  Algorithm  or  the  Modified  Levinson's  Algorithm  allows 
the  solution  of  a Hermitian  Topelitz  system  of  linear  equations 
to  be  obtained  with  substantially  less  computational  effort 
than  that  required  by  standard  non-iterative  methods.  In  fact, 
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the  modified  Levinson's  Algorithm  can  be  used  to  obtain  the 
exact  solution  using  approximately  twice  the  number  of  opera- 
tions required  to  carry  out  a single  step  of  a fixed-point 
iterative  technique  such  as  Jacobi  Relaxation  or  Successive 

A 

Over  Relaxation.  However,  the  restriction  that  4>  be  Topelitz 

AA 


makes  Levinson's  Algorithm  less  general  than  fixed-point  itera- 
tion techniques . 

4.2  Adaptive  Estimation  Using  Levinson's  Algorithm 

The  computational  efficiency  of  the  Levinson  Algorithm 
makes  it  attractive  for  adaptive  estimation  when  the  estimated 

A 

autocorrelation  matrix  has  a Topelitz  structure  .as  does  the 

. . 2 
estimate  proposed  xn  Section  3.1.  However,  the  0(N  ) operations 

required  by  Levinson ' s Algorithm  may  be  restrictive  in  many 

applications.  In  this  case,  a procedure  analogous  to  the 

Variable  Iteration  Step  Method  for  fixed-point  iteration  can  be 

used.  Specifically,  the  solution  to  the  system  (3.26)  can  be 

obtained  by  the  use  of  Levinson's  Algorithm  every  k samples  while 

/s  /\  j 

the  estimates  4>xx  and  R^  are  updated  at  each  sample  time  n. 

Under  this  scheme,  the  average  number  of  operations  per  sample 
required  by  Levinson's  Algorithm  can  be  reduced  by  a factor  of 
k.  The  exact  average'  number  of  operations  per  sample  required 
by  Levinson's  Algorithm  is  given  in  Figure  4.3. 

Figure  4.3  gives  the  average  number  of  operations  per 
sample  required  by  both  forms  of  Levinson's  algorithm.  How- 
ever, to  determine  the  average  number  of  operations/sample 
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required  by  the  adaptive  estimation  procedure,  the  number  of 
operations  needed  to  compute  the  estimated  autocorrelation 
matrix  and  vector  must  be  added  to  the  average  number  of  opera- 
tions/sample required  by  the  Levinson  algorithm.  If  the  estimate 


Algorithm 

Additions 

Multiplications 

Divisions 

Levinson 

£(2.5N2+0.5N-3) 

|(2.5N2-2.5N) 

|(4N-2) 

Modified 

|-(2N2+N-2) 

i(2N2+2N-4) 

£(4N-2) 

Figure  4.3  Average  Number  of  Operations/Sample  Required 

Levinson's  Algorithm  and  the  Modified  Levinson 
Algorithm  when  carried  out  over  k sample  periods. 

proposed  in  Section  3.1  is  used  in  conjunction  with  Levinson's 
Algorithm  to  accomplish  the  adaptive  estimation  task,  then  the 
required  average  number  of  operations/sample  can  be  obtained  by 
adding  the  number  of  operations  required  to  estimate  $xx  and 
using  (3.25)  to  the  entries  of  Figure  4.3.  The  number  of 

be  found  in  Figure  3.1.  The  total  average  number  of  operations/ 
sample  required  for  the  solution  of  (3.26)  using  Levinson's 

A 

Algorithm  applied  every  k sample  periods  and  updating  $xx  and 

A 

R^  every  sample  period  is  given  in  Figure  4.4. 

Comparing  Figures  4.3  and  3.4,  it  can  be  seen  that  the 
use  of  the  Levinson  Algorithm  produces  the  exact  solution  to  the 
system  (3.26)  with  an  amount  of  computational  effort  which  is  on 
the  same  order  as  that  required  to  carry  out  a single  step  of 


operations  required  to  estimate  4>xx  and  R^  using  (3.25)  can 


the  Successive  Over-Relaxation  Method.  Thus,  Levinson's  Al- 
gorithm is  attractive  for  adaptive  estimation  when  the  estimated 
autocorrelation  matrix  has  a Topelitz  structure. 


4.3  Conclusion 


Levinson's  Algorithm  is  an  attractive  approach  to  the  solu-- 
tion  of  the  system  of  linear  equations  (3.26)  in  that  it  deter- 
mines the  exact  solution  very  efficiently  in  a non-iterative 
fashion.  Using  a procedure  similar  to  the  Variable -Iteration - 
Step  Method,  the  average  number  of  operations/sample  can  be 
reduced  by  solving  the  system  using  Levinson's  Algorithm  every 
k sample  periods  while  updating  the  estimated  autocorrelation 
matrix  and  vector  at  each  sample  time.  This  method  gives  a 
flexible  approach  to  the  adaptive  estimation  problem  in  that  full 
utilization  of  the  time  available  between  samples  can  be  made 
for  computation. 

Unfortunately,  the  Levinson  Algorithm  requires  that  the 
autocorrelation  matrix  estimate  have  a Topelitz  structure. 

A 

While  the  estimate  4>xx  proposed  in  Section  3.1  has  such  a struc- 
ture, other  estimates  of  interest  may  not.  In  this  respect,  the 
Adaptive  Fixed-Point  Iteration  Method  is  superior  in  that  it  is 
applicable  to  a more  general  class  of  matrices.  In  addition, 
the  Levinson  Algorithm  is  recursive  in  nature  and  not  well 
suited  to  parallel  processing  as  are  the  two  fixed-point  itera- 
tion methods  discussed  in  the  previous  chapter.  Suitability  of 
aun  algorithm  for  parallel  processing  is  of  extreme  importance 
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when  real-time  applications  which  involve  large  values  of  N 
are  considered. 

Levinson's  Algorithm  offers  a viable  approach  to  adaptive 
estimation  for  the  somewhat  restrictive  class  of  problems  to 
which  it  is  applicable.  It  is  very  attractive  in  situations  in 
which  the  sample  period  is  large  enough  so  that  a large  number 
of  computations  cam  be  carried  out  between  samples  without  the 
use  of  parallel  processing.  The  performance  of  the  adaptive 
estimation  method  using  Levinson's  Algorithm  described  in  this 
chapter  will  be  compared  to  that  of  the  Adaptive  Fixed-Point 
I Iteration  methods  proposed  in  Chapter  III  and  to  that  of  the  LMS  i 


Gradient  Method  via  computer  simulation  in  Chapter  VI. 


CHAPTER  V 


THEORETICAL  PERFORMANCE  OF  THE  LMS  GRADIENT  ALGORITHM 
AND  ADAPTIVE  FIXED-POINT  ITERATION 


Both  the  LMS  Gradient  Algorithm  and  the  Adaptive  Fixed- 
Point  Iteration  procedure  proposed  in  Chapter  3 are  iterative 
methods  for  the  approximate  solution  of  the  discrete  finite 
impulse  response  Wiener-Hopf  equation  given  by: 


$XX  - “ -XX 


(1.25) 


Since  these  methods  are  iterative  in  nature,  their  rates  of 
convergence  are  of  importance.  Theoretical  bounds  on  the  rates 
of  convergence  in  the  mean  of  the  LMS  Gradient  Algorithm,  the 
Jacobi  Relaxation  Method  and  the  Successive  Over-Relaxation  Method 

T 

were  derived  in  Chapters  II  and  III.  While  only  approximate, 
these  bounds  are  useful  for  comparing  convergence  rates  of  the 
LMS  Gradient  Method  over  variations  in  the  parameter  u and  of 
the  Jacobi  Relaxation  and  SOR  Methods  over  variations  in  the 
relaxation  parameter  0 . In  addition , these  bounds  are  useful 
for  evaluating  the  Jacobi  Relaxation  and  SOR  Algorithms  when  they 
are  used  with  the  Variable  Iteration  Method  and  for  making 
general  comparisons  of  the  rates  of  convergence  of  the  three 
iterative  algorithms. 
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In  this  chapter,  the  rates  of  convergence  of  the  LMS 
Gradient  Algorithm,  the  Jacobi  Relaxation  Method  and  the  SOR 
Method  are  studied  for  a particular  problem  under  varying 
conditions.  The  term  "rate  of  convergence",  as  used  in  this 
chapter,  will  refer  to  the  rate  of  convergence  of  the  mean- 
square  error  produced  by  the  filter  determined  by  a particular 
algorithm  to  the  minimum  MSE  which  would  be  obtained  by  the  use 
of  an  optimal  filter.  The  derivation  of  the  expressions  used  in 
computing  the  bounds  considered  in  this  chapter  are  given  in 
Sections  2.3,  3.3.2  and  3.4.1.  Comparisons  between  the  theoreti 
cal  bounds  and  "actual"  performance  will  be  investigated  in 
Chapter  VI  by  the  use  of  computer  simulation. 


5.1  Statement  of  the  Problem 


The  theoretical  bounds  on  the  rates  of  convergence  of  the 
LMS  Gradient  Algorithm,  the  Jacobi  Relaxation  Method  and  the  SOR 
.Method  derived  in  Chapters  II  and  III  require  knowledge  of  the 

input  autocorrelation  matrix  if  they  are  to  be  used  in  the  study 
of  the  convergence  of  these  iterative  algorithms.  Therefore, 
some  particular  problem  must  be  selected  for  which  the  auto- 
correlation matrix  can  be  determined  a priori.  Sinusoidal  sig- 
nals comprize  an  important  class  of  inputs  for  a Spectral  Line 
Enhancer  due  to  their  widespread  use.  In  addition,  the  auto- 
correlation function  of  a sinusoidal  signal  does  not  decay  to 
zero  as  the  amount  of  delay  increases.  Therefore,  the  sinusoid 
is  the  limiting  case  of  a coherent  signal.  In  the  problem 
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considered  in  this  chapter,  the  narrowband  input  to  the  Spectral 

i 

Line  Enhancer  is  assumed  to  be  a single  sinusoid  of  known  amp- 
litude and  frequency  but  of  unknown  phase.  This  problem  is  of- 

| 

ten  referred  to  as  the  "signal  known  except  phase"  (SKEP)  problem.; 
It  will  be  assumed  here  that  the  phase  is  uniformly  distributed 
1 on  the  interval  [ — tt  , tt ] . Therefore,  in  the  problem  considered 
here,  the  signal,  or  narrowband  input  to  the  Spectral  Line 
Enhancer  will  be  assumed  to  have  the  form: 

S(n)  = A cos(£2n+9)  (5.1) 

| where:  A = Signal  Amplitude 

j 

ft  = Signal  Frequency  (Radians) 

j 

n = Time  Index 

0 = Random  Phase,  uniformly  distributed  on 

The  Spectral  Line  Enhancer  assumes  that  one  component  of 
the  data  is  relatively  narrowband  compared  to  the  other  compon- 

I 

ent.  Therefore,  the  noise  process  chosen  for  this  problem  must 
be  relatively  wideband  compared  to  the  signal  given  in  (5.1). 

In  other  words,  the  autocorrelation  function  of  the  noise  must 
be  essentially  zero  for  delays  of  n^  or  greater.  To  satisfy  this 
assumption  for  the  problem  considered,  an  autoregressive  noise 
sequence  was  chosen  as  the  process  to  be  added  to  the  signal. 

I 

This  type  of  noise  sequence  may  be  generated  by  the  following 
j relationship: 
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N(n)  = Z a y(n-i)  (5.2) 

i=0 

I 

where  y(i)  are  independent  identically  distributed  random  varia- 
bles  with  zero  mean  and  variance  = a , and  |a|  < 1 is  the  auto- 

regression parameter.  As  will  be  shown  later,  N(n)  generated 
by  using  (5.2)  has  an  autocorrelation  function  which  is  zero  for 
delays  of  n1  or  greater  which  satisfies  the  assumption  made  in 
the  formulation  of  the  Spectral  Line  Enhancer. 

Thus,  the  SKEP  problem  considered  in  this  chapter  is  that 
of  a single  sinusoidal  signal  given  by  (5.1)  in  the  presence  of 

an  additive  autoregressive  noise  sequence  generated  according  to 

* 

(5.2).  If  it  is  assumed  that  the  signal  and  the  noise  are 
orthogonal , then  the  determination  of  the  input  autocorrelation 
matrix  is  straight  forward  as  will  be  shown  in  the  next  section. 

I 

5.1.1  Determination  of  the  Autocorrelation  Matrix 

I 

In  order  to  investigate  the  convergence  properties  of  the 
LMS  Gradient  Algorithm,  the  Jacobi  Relaxation  Method  and  the  SOR 
Method  using  the  theoretical  bounds  derived  in  Chapters  II  and 
III,  it  is  necessary  to  determine  the  input  autocorrelation  matrix 
for  the  problem  considered.  In  the  case  of  the  SKEP  problem 
considered,  the  determination  of  $xx  is  fairly  straightforward. 

I 

However,  the  derivation  will  be  included  here  for  completeness. 

Each  element  of  the  input  autocorrelation  matrix  $xx  can 
be  written  as  follows : 
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♦ ^ - E[X(i)X(j)  ) . 


(5.3) 


However,  since  the  signal  sequence  and  the  noise  sequence  are 
assumed- to  be  orthogonal,  (5.3)  can  be  rewritten  as: 


♦ ij  = E[S(i)S(j)  ] + E [N  (i) N ( j ) ] 


(5.4) 


Therefore,  determination  of  the  matrix  <J>XX  can  be  reduced  to  the 


computation  of  two  autocorrelation  functions,  Rgg(i,j)  and 


RNN(i,j),  where  the  following  definitions  apply: 


Rss  (i,  j ) = E[S(i)S(j)  ] 


R^U/j)  = E [N  (i)  N ( j ) ] 


(5.5) 

(5.6) 


The  autocorrelation  function  of  the  signal  can  be  obtained 
by  substituting  for  S(i)  and  S(j)  in  (5.5)  from  equation  (5.1). 
Equation  (5.5)  can  then  be  written  as: 


Rgg(i,j)  = A E [cos (fti+0) cos (ft j+0) ] 


(5.7) 


Equation  (5.7)  can  be  reduced  using  a well-known  trigonometric 
identity  to: 


Rgg  (i,  j ) = ^ — E [cos  (0  (i-j  ) ) +cos (fi  (i+j  ) +2  0 ) ] . (5.8) 


However,  the  term  cos(ft(i-j))  is  not  a random  quantity.  There- 


fore,  (5.8)  reduces  to: 


RgS  (i,  j)  = j-  cos(fi(i-j) ) + y~  E[coS(fl(i+j)+20) ] . (5.9) 

Now,  it  is  assumed  that  the  phase  0 was  distributed  uniformly 
on  the  interval  [ — it , tt ] . Therefore,  the  expected  value  term  in 
(5.9)  is  defined  as: 

1 17  • 

E[cos  (n(i+j)+20)  ] = yjjr  / cos (fi (i+j ) +20) d0  . (5.10) 

— TT 

But,  the  integral  in  the  above  expression  is  equal  to  zero. 

Thus : 

E [cos (Q (i+j ) +20) ] = 0 

Therefore,  the  autocorrelation  function  of  the  signal  can  be 
expressed  as: 


Rss(i,j)  - cos(fl(i-j))  . (5.11) 

It  is  obvious  from  (5.11)  that  the  autocorrelation  function  of 
the  signal  for  the  SKEP  problem  is  a stationary  statistic  since 
it  is  a function  only  of  (i-j).  Therefore,  if  RNN(i,j)  is  also 
a stationary  statistic,  then  the  input  autocorrelation  matrix 
will  exhibit  a Topelitz  structure. 

The  autocorrelation  function  for  the  autoregressive  noise 
sequence  can  be  determined  by  considering  equation  (5.6)  and 
substituting  for  N(i)  and  N(j)  from  (5.2).  Equation  (5.6)  can 
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: 


then  be  written  as: 

. 

n^“l  ^l"”^ 

E [N  (i)  N ( j ) ] = E [ ( £ a 1 Y(i-M)(  l ak  Y(j-k))].  (5.12) 

1=0  k=0 

Expanding  (5.12)  yields: 


Therefore,  as  required  by  the  assumption  made  in  the  formulation 
of  the  Spectral  Line  Enhancer,  noise  samples  which  are  separated 


by  delays  of  n^  or  greater  are  orthogonal.  In  the  case  where 
(i- j ) = (i-k)  , R^dfj)  can  be  determined  by  making  the  substitu- 
tion m*|i-j|.  With  this  substitution,  (5.13)  can  be  rewritten  as 

"r1 

E [N  (i ) N ( j ) ] - I a2Z-mE[Y(i-Jl)Y(i-Jl)  ] . (5.16) 

£=m 

. 2 
Now,  since  Y(i)  was  assumed  to  have  zero  mean  and  variance  a , 

(5.16)  can  be  rewritten  as: 

2 (ni~| i- j 

E [N  (i)  N(  j ) ] = a2  — » . (5.17) 

1-a 


Thus,  combining  the  results  of  (5.15)  and  (5.17)  the  autocorrela- 
tion function  of  the  noise  can  be  expressed  as : 


E[N(i)N(j)]  = a2 


2 (n;L-|i-j  | ) 


i- j I < n. 


1-a 


= 0 otherwise  . (5.18) 

Equations  (5.11)  and  (5.18)  can  be  substituted  into  (5.4) 
to  determine  the  autocorrelation  matrix  of  the  input  data.  It 
is  obvious  from  (5.11)  and  (5.18)  that  $xx  will  have  a Topelitz 
structure  since  both  Rss(i,j)  and  ( i , j ) are  functions  only  of 
(i-j).  Once  4>  has  been  determined,  the  theoretical  bounds 

AA 

derived  in  Chapters  II  and  III  can  be  used  to  study  the  converg- 
ence properties  of  the  three  iterative  algorithms  considered. 


’;~.v  vrifflhaf 


5.2  Theoretical  Performance  of  the  LMS  Gradient  Algorithm 

A theoretical  bound  on  the  rate  of  convergence  of  the  mean 
square  error  produced  by  the  filter  determined  by  the  LMS 
Gradient  Algorithm  was  derived  in  Section  2,3  and  was  given  by 
the  relationship  in  (2.49).  As  discussed  in  Section  2.3,  the 
rate  of  convergence  of  the  MSE  to  the  minimum  MSE  produced  by 
the  optimal  filter  depends  on  the  parameter  y present  in  the  LMS 
Gradient  Algorithm.  In  this  section,  the  rate  of  convergence 
of  the  LMS  Gradient  Algorithm  for  the  SKEP  problem  is  explored 
for  various  values  of  the  parameter  y under  several  different 
signal-to-noise  ratio  (SNR)  conditions.  The  SNR  is  defined  here 
as  the  ratio  of  the  signal  variance  to  the  noise  variance  on  a 
per  sample  basis.  That  is,  SNR  is  defined  as: 


The  rates  of  convergence  for  the  different  values  of  y are  stud- 
ied using  the  theoretical  bound  given  by  (2.49)  and  therefore 
"convergence"  is  interpreted  as  convergence  in  the  mean. 

Figure  5.2  shows  computer-generated  plots  of  the  theoreti- 
cal convergence  bound  given  in  (2.49)  for  the  LMS  Gradient  Al- 
gorithm for  various  values  of  the  parameter  y.  Each  plot  is  for 
a different  condition  of  signal-to-noise  ratio  with  the  four 
curves  in  each  plot  corresponding  to  four  different  values  of  y. 
The  vertical  axis  in  each  figure  is  the  MSE  axis,  while  the 
horizontal  axis  corresponds  to  the  number  of  sample  periods.  The 
values  of  y which  correspond  to  each  of  the  four  curves  in  each 
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Figure  5.2  Theoretical  Performance  of  the  LMS  Gradient 

Algorithm  for  Various  Values  of  the  Parameter 
u Under  Four  Different  SNR  Conditions. 
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of  the  figures  are  values  which  might  reasonably  be  used  in 
practice.  They  have  been  selected  so  that  there  is  a factor  of 
four  difference  in  the  value  of  y between  any  two  adjacent 

i 

curves  in  each  of  the  four  plots.  As  can  be  seen  from  the  fig- 
ures, the  value  of  the  parameter  y has  a significant  effect  on 
the  rate  of  convergence  of  the  LMS  Gradient  Algorithm.  Specifi- 
cally, a decrease  in  the  value  of  y greatly  increases  the  num- 
ber of  sample  periods  required  for  the  LMS  Gradient  Algorithm 
to  converge.  The  theoretical  bounds  considered  here  will  be 
compared  to  the  "actual"  convergence  properties  of  the  LMS 
Gradient  Algorithm  via  computer  simulation  in  Chapter  VI. 

5.3  Theoretical  Performance  of  the  Jacobi  Relaxation  Method 

Equation  (3.75),  derived  in  Section  3.3.2,  gives  a theo- 
retical bound  on  the  rate  of  convergence  of  the  mean-square 
error  produced  by  the  filter  determined  by  the  Jacobi  Relaxation  i 
Method  to  the  minimum  MSE  which  would  be  obtained  by  the  use  of 
the  optimal  filter.  As  discussed  in  Section  3.3.2,  the  relaxa- 
tion parameter  8 directly  effects  the  rate  of  convergence  of  the 
Jacobi  Relaxation  Method.  In  this  section,  the  rate  of  converg- 
ence of  the  MSE  produced  by  the  filter  determined  by  the  Jacobi 
Relaxation  Method  to  the  MSE  produced  by  the  optimal  filter 
for  the  SKEP  problem  is  studied  for  various  values  of  the  relaxa- 
tion parameter  8 under  several  different  conditions  of  input 
signal-to-noise  ratio.  Signal-to-noise  ratio  is  defined  as  in 
(5.19).  In  addition,  the  theoretical  rate  of  convergence  of 


I 


L 
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Jacobi  Relaxation  operating  under  the  Variable  Iteration  Step 

I 

Method  is  also  studied  for  various  SNR  conditions  in  this  sec- 
tion. In  all  cases,  the  theoretical  convergence  curves  were 

i 

generated  using  equation  (3.75)  as  derived  in  Section  3.3.2. 

As  was  the  case  for  the  LMS  Gradient  Method , convergence  of 
produced  by  Jacobi  Relaxation  to  the  optimal  solution  is  defined 
as  convergence  in  the  mean. 

| 

5.3.1  Theoretical  Performance  of  Jacobi  Relaxation  for  Varying  3 
As  stated  previously,  the  rate  of  convergence  of  Jacobi 
Relaxation  depends  directly  on  the  value  of  the  relaxation  para- 
meter 3 . To  investigate  the  rate  of  convergence  with  various 
values  of  8 for  the  SKEP  problem,  theoretical  convergence  curves 
were  generated  by  a computer  according  to  equation  (3.75). 

Figure  5.3  shows  the  resulting  curves.  Each  plot  in  the  figure 
corresponds  to  a different  signal-to-noise  ratio  condition  with 
each  curve  in  a plot  corresponding  to  a different  value  of  the 
relaxation  parameter  8 . The  values  of  3 chosen  are  such  that 
there  is  a factor  of  four  difference  in  the  value  of  8 between 
any  two  adjacent  curves.  The  largest  value  of  8=0.14,  corres- 
ponding to  the  left-most  curve  in  each  figure  was  chosen  by  apply 
ing  the  bound  on  8 given  in  equation  (3.72).  This  bound  was 
used  since  its  application  does  not  require  knowledge  of  the 
statistics  of  the  input  process  which  would  not  be  available  in 
practice.  The  vertical  axis  for  each  plot  is  the  MSE  axis  while 
the  horizontal  axis  gives  the  number  of  sample  periods. 
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Note  from  Figure  5.3  that  the  value  of  the  relaxation 
parameter  8 has  a significant  effect  on  the  rate  of  convergence 
of  the  Jacobi  Relaxation  Method.  Specifically,  decreasing  the 
value  of  8 slows  the  rate  of  convergence  or  increases  the  number 
of  sample  periods  required  for  the  MSE  produced  by  the  filter 
determined  by  the  Jacobi  Relaxation  Method  to  reach  the  minimum 
MSE  produced  by  the  optimal  filter.  The  theoretical  bounds 
considered  here  will  be  compared  to  the  "actual"  performance 
of  the  Jacobi  Relaxation  Method  using  computer  simulation  in 
Chapter  VI . 


5.3.2  Theoretical  Performance  of  Jacobi  Relaxation  Under  the 
Variable "Iteration  Step  Method 

2 

The  Jacobi  Relaxation  Method  requires  on  the  order  of  N 
computations  per  iteration  step  when  implemented  directly  using 
equation  (3.76)  as  can  be  seen  in  Figure  3.2.  The  Variable 
Iteration  Step  Procedure  was  proposed  in  Section  3.5.1  as  a 
method  for  reducing  the  average  number  of  operations  per  sample 
for  the  Fixed-Point  Iteration  Algorithms.  In  this  procedure, 
the  Jacobi  Relaxation  Method  was  used  to  update  & elements  of  the 
unit  sample  response  vector  every  k samples.  Therefore,  if 
Jt/k  < N,  the  average  number  of  operations/sample  can  be  reduced 


Vk  (N+l)  < N (N+l) 


(5.20) 
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Figure  5.3  Theoretical  Performance  of  the  Jacobi  Relaxation 
Algorithm  for  Various  Values  of  the  Parameter 
8 Under  Four  Different  SNR  Conditions. 
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However,  as  mentioned  in  Section  3.5.1,  such  a reduction  in  the 
average  number  of  operations  per  sample  is  accompanied  by  a 
corresponding  increase  in  the  number  of  sample  periods  required 
for  convergence . Figure  5 . 4 shows  computer-generated  plots  giv- 
ing the  theoretical  bounds  on  the  rate  of  convergence  of  the  MSE 
produced  by  the  filter  determined  by  the  Jacobi  Relaxation 
Algorithm  operating  under  the  Variable  Iteration  Step  Method. 


Each  plot  in  the  figure  corresponds  to  a different  SNR  condition 
for  the  SKEP  problem  and  each  curve  in  a plot  corresponds  to  a 
different  value  of  1/ k.  In  all  the  plots,  a curve  corresponding 
to  the  ordinary  Jacobi  Relaxation  Method  (Jl=N,  k=l)  is  included 
as  a reference.  In  all  cases,  the  relaxation  parameter  8 was 
* chosen  to  be  8=0.14  which  was  determined  from  the  bound  given  in 
equation  (3.72)  as  discussed  in  the  last  section.  The  value  of  j 

i 

& in  the  Variable  Iteration  Step  Method  has  been  fixed  at  £=N=16 
since  the  FFT  Method  proposed  for  the  implementation  of  Jacobi 
Relaxation  updates  all  N elements  of  h at  each  iteration  step. 

In  all  cases,  the  vertical  axis  is  the  MSE  axis  while  the  hori- 
zontal axis  gives  the  number  of  sample  periods.  j 

As  can  be  seen  in  Figure  5.4,  decreasing  the  ratio  1/ k 
causes  a corresponding  increase  in  the  number  of  sample  periods 
required  for  convergence.  The  increase  is  directly  proportional 
to  the  parameter  Jl/k.  The  theoretical  bounds  considered  here  will 
be  compared  to  the  "actual"  performance  of  the  Jacobi  Relaxation  j 
Algorithm  operating  under  the  Variable  Iteration  Step  Method  in 
Chapter  VI  using  computer  simulation. 
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5.4  Theoretical  Performance  of  the  SOR  Method 

A theoretical  bound  on  the  convergence  of  the  mean-square 
error  produced  by  the  filter  determined  by  the  SOR  Method  was 
derived  in  Section  3.4.1  and  is  given  by  equations  (3.91)  and 
(3.92).  As  can  be  seen  from  these  equations,  the  relaxation 

i 

parameter  8 effects  the  rate  of  convergence  of  the  SOR  Method. 

In  this  section,  the  rate  of  convergence  of  the  MSE  produced 
by  the  filter  determined  by  the  SOR  Method  to  the  minimum  MSE 
which  would  be  obtained  by  the  optimal  filter  for  the  SKEP  prob- 
lem is  studied  for  various  values  of  the  relaxation  parameter  8 
under  several  different  signal-to-noise  ratio  conditions.  In 
addition,  the  theoretical  rate  of  convergence  of  the  MSE  of  the 
filter  determined  by  the  SOR  Method  operating  under  the  Variable 
Iteration  Step  scheme  is  considered  for  various  SNR  conditions. 

In  all  cases  the  theoretical  bounds  were  generated  using  equa- 
tions (3.91)  and  (3.92)  and  convergence  of  h^  produced  by  the 
SOR  Method  to  the  optimal  solution  is  interpreted  as  convergence 
in  the  mean. 

5.4.1  Theoretical  Performance  of  Successive  Over-Relaxation 
for  Varying  8 

The  rate  of  convergence  of  the  SOR  Method  depends  directly 
on  the  value  of  the  relaxation  parameter  8 as  discussed  previous - 
Theoretical  convergence  curves  were  generated  for  the  SKEP 
» t rjy  ".he  ase  of  equations  (3.91)  and  (3.92)  to  investigate 
••  ••••  * - -mr  }« rce  of  the  MSE  produced  by  the  filters  de- 

• •»  so*  Method  w th  various  values  of  3 . The  result- 
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ing  curves  are  shown  in  Figure  5.5.  Each  plot  in  the  figure 
corresponds  to  a different  signal-to-noise  ratio  condition  with 
each  curve  in  a plot  corresponding  to  a different  value  of  the 
relaxation  parameter  8.  Each  of  the  three  values  of  8 chosen  lie 
within  the  range  0 <8  <2  which  is  required  for  convergence  of 

the  SOR  Method.  Figure  5.17  gives  the  SNR  condition  and  values 
j of  8 which  correspond  to-  each . figure . The  vertical  axis  for  each 
plot  is  the  MSE  axis  while  the  number  of  sample  periods  is  along 
the  horizontal  axis. 

Note  from  Figure  5.5  that  the  value  of  the  relaxation 
parameter  8 has  a significant  effect  on  the  rate  of  convergence 
of  the  SOR  Method.  However,  as  can  be  seen  from  the  figures, 
the  relationship  between  the  rate  of  convergence  and  the  value 
of  8 is  not  straightforward  as  was  the  case  with  the  Jacobi 
Relaxation  Method.  In  fact,  except  in  certain  special  cases  it 
is  extremely  difficult  to  determine  the  value  of  6 which  is  most 
advantageous  for  convergence  [20].  The  "actual"  performance  of 
the  SOR  Method  as  determined  by  computer  simulation  will  be  com- 
pared to  the  theoretical  bounds  considered  here  in  Chapter  VI. 

] 

5.4.2  Theoretical  Performance  of  Successive  Over-Relaxation 

Under  the  Variable  Iteration  Step  Method 

The  implementation  of  the  SOR  Method  using  equation 

2 

(3.93)  requires  on  the  order  of  N computations  per  iteration 
step  as  can  be  seen  in  Figure  3.2.  The  Variable  Iteration  Step 

I 

Procedure  proposed  in  Section  3.5.1  is  a method  for  the  reduction 
of  the  average  number  of  operations  per  sample  required  by  the 
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SOR  Method.  In  this  method,  the  SOR  Algorithm  was  used  to  up- 

I 

date  & elements  of  the  unit  sample  response  vector  every  k sam- 
ple  periods.  Therefore,  if  l/k  <N,  then  the  average  number  of 
operations/sample  can  be  reduced  to: 

Z/k  (N+l)  . < N(N+1)  . (5.21) 

Obviously,  such  a reduction  in  the  average  number  of  operations 
per  sample  is  accompanied  by  a corresponding  increase  in  the 
number  of  sample  periods  required  for  convergence.  To  investi- 
gate the  performance  of  the  SOR  Method  with  variable  iteration 
steps,  theoretical  convergence  curves  were  generated  using  equa- 
tions (3.91)  and  (3.92).  The  resulting  curves  are  given  in  Fig- 
ure 5.6.  Each  plot  in  the  figure  corresponds  to  a different 
condition  of  signal-to-noise  ratio  with  each  curve  in  a plot 
corresponding  to  different  values  of  Z and  k.  A curve  for  the 
ordinary  SOR  Method  (Jt=N,  k=l)  has  been  included  in  each  figure 
as  a reference.  In  all  cases,  the  vertical  axis  is  the  MSE  axis 
and  the  horizontal  axis  gives  the  number  of  sample  periods. 

! 

As  can  be  seen  from  Figure  5.6,  the  values  of  Z and  k 
chosen  for  the  variable  iteration  step  method  have  a significant 
effect  on  the  rate  of  convergence  of  the  SOR  Algorithm.  Each 
plot  in  tiie  figure  contains  three  curves.  The  curve  which  ex- 
hibits the  highest  rate  of  convergence  is  the  reference  curve 
corresponding  to  Z= 16  and  k=l  as  expected.  The  remaining  two 
curves  correspond  to  (£,k)  values  which  were  chosen  so  that  the 
ratio  Z/k  would  equal  one.  In  this  case,  the  average  number  of 


operations  per  sample  for  the  SOR  Method  is  on  the  order  of  N, 
the  length  of  the  filter.  As  can  be  seen  from  the  figure,  the 
curve  for  the  case  (1*1,  k=l)  exhibits  a higher  rate  of  con- 
vergence than  the  case  where  (1=16,  k=16)  since  its  convergence 
curve  always  lies  below  that  for  the  case  (1=16,  k=16) . The 
theoretical  bounds  considered  here  will  be  compared  to  the 
"actual"  performance  of  the  SOR  Algorithm  operating  under  the 
Variable  Iteration  Step  Method  in  Chapter  VI  by  the  use  of 
computer  simulation. 


5.5  Comparison  of  the  Three  Algorithms 


In  the  previous  sections,  the  convergence  properties  of  the 
LMS  Gradient  Algorithm,  the  Jacobi  Relaxation  Method  and  the  SOR 
Algorithm  were  studied  under  varying  conditions  using  the  theo- 
retical bounds  derived  in  Chapters  II  and  III.  While  these  bounds 
are  only  approximate  and  neglect  covariance  terms,  comparisons 
among  the  three  algorithms  can  still  be  made.  In  this  section, 
theoretical  convergence  curves  for  the  three  iterative  algorithms 
will  be  compared.  The  "actual"  convergence  of  the  algorithms 
will  be  studied  via  computer  simulation  in  Chapter  VI. 

The  LMS  Gradient  Algorithm  requires  fewer  operations  per 
sample  than  the  two  fixed-point  iteration  algorithms.  Also, 
the  estimate  of  the  input  statistics  used  by  the  LMS  Gradient 
Algorithm  has  a larger  variance  than  the  estimate  used  by  the 
Adaptive  Fixed-Point  Iteration  Algorithms.  Therefore,  any  mean- 
ingful comparison  of  the  three  algorithms  should  attempt  to 
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equalize  the  number  of  operations/sample  and  the  variance  of  the 
estimate  of  the  statistics  across  the  three  algorithms.  The 

■ 

average  number  of  operations/sample  can  be  equalized  approximate-  I 
ly  by  the  use  of  the  Variable  Iteration  Step  Method  with  the 
Jacobi  Relaxation  Method  and  the  SOR  Algorithm.  Under  the  Var- 
iable Iteration  Step  Method,  the  average  number  of  operations/ 
sample  required  by  the  fixed-point  iteration  algorithms  can  be 
reduced  to  0(N).  Thus,  in  the  comparisons  that  follow,  the 
ratios  1/ k are  chosen  to  be  one  for  both  the  Jacobi  Relaxation 
and  the  SOR  Methods.  For  the  Jacobi  Relaxation  Method,  i and  k 
are  both  chosen  to  be  N,  while  for  the  SOR  Method,  i and  k are 
both  chosen  to  be  1.  Therefore,  the  average  numbers  of  opera- 
tions/sample are  on  the  order  of  N for  all  three  iterative  al- 
gorithms considered.  However,  the  Adaptive  Fixed-Point  Iteration  i 
methods  still  require  more  operations/sample  than  the  LMS  Grad- 
ient Algorithm  due  to  the  0(N)  operations/sample  required  to 
estimate  the  autocorrelation  matrix  and  autocorrelation  vector. 

The  estimate  of  the  statistics  of  the  input  process  used 

I 

by  the  LMS  Gradient  Method  is  given  by  (2.17)  as: 

I 

VE[e2(n)|h]  = 2 (S  (n)  -X  (h)  ) X (n-n^  . (2.17) 

A 

Substituting  for  S(n),  this  estimate  of  the  gradient  of  the  MSE 
can  be  rewritten  as : 


Ve [e2 (n) | h]  = 2[X(n-n1)XT(n-n1)hn-X(n)X(n-n1) ] (5.22) 


which  may  also  be  expressed  as: 


where : 


vEte2(n)|hl  - 2(*xx  h,,  - 


^XX  = X(n-n1)XA (n-n1) 


(5.23) 


(5.24) 


= X(n)X(n-n1) 


(5.25) 


The  estimates  4>xx  and  given  in  (5.24)  and  (5.25)  are  similar 
tothe  estimates  of  the  autocorrelation  matrix  and  vector  used 
in  the  two  Adaptive  Fixed-Point  Iteration  Algorithms.  Therefore, 

A A 

the  variance  of  each  element  of  4>xx  and  R^  is  given  by  (3.22) 
with  M=l.  A comparison  of  the  LMS  Gradient  Algorithm  and  the 
Jacobi  Relaxation  Algorithm  reveals  that  the  same  basic  iteration 
relationship  is  used  in  both  cases.  This  iteration  relationship 
is  given  by: 


^n+1  C($XX  ^n  " ^XX} 


(5.26) 


The  difference  between  the  two  algorithms  lies  in  the  choice  of 

A A 

the  parameter  C and  the  estimates  $xx  and  R^.  In  the  compari- 
sons which  follow,  the  value  of  S for  the  Jacobi  Relaxation  Al- 
gorithm was  chosen  according  to  the  bound  given  by  equation 
(3.72).  The  parameter  y for  the  LMS  Gradient  Algorithm  was  then 

A 

chosen  so  that  the  variance  of  each  term  of  the  matrix  2u<J>xx 

A 

and  the  vector  2yRxx  was  equal  to  the  variance  of  the  correspond- 
ing matrix  and  vector  in  the  Jacobi  Relaxation  Algorithm  for  a 
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data  window  of  length  M=512.  Since  for  large  M the  variance  of 
the  estimates  used  in  the  Adaptive  Fixed-Point  Iteration  Algor- 
ithms decrease  at  a rate  of  approximately  1/M,  y was  selected  to 


>r^To: 


v612 


(5.27) 


for  all  the  comparisons  considered  in  this  section.  Equation 
(5.27)  yields  values  for  y which  might  realistically  be  used  in 
practice  for  all  the  SNR  conditions  considered  here. 

Figure  5 . 7 gives  the  comparisons  of  the  theoretical  con- 
vergence curves  for  the  LMS  Gradient  Algorithm,  the  Jacobi  Re- 
laxation Method  and  the  SOR  Method.  Each  plot  in  the  figure 
corresponds  to  a different  SNR  condition  for  the  SKEP  problem. 

In  all  cases,  the  vertical  axis  is  the  MSE  axis  while  the  number 
of  sample  periods  is  displayed  along  the  horizontal  axis. 

It  can  be  seen  from  Figure  5.7(A)  that  for  the  high  SNR 
condition,  the  three  algorithms  exhibit  approximately  the  same 
rate  of  convergence.  However,  for  the  lower  SNR  conditions  the 
SOR  Method  exhibits  a significantly  better  convergence  rate  as 
can  be  seen  in  Figures  5.7(B)  through  5.7(D).  The  convergence 
rate  of  the  Jacobi  Relaxation  Method  is  slightly  greater  than 
that  of  the  LMS  Gradient  Algorithm  at  the  lower  signal-to-noise 
ratios.  Thus,  it  appears  from  these  theoretical  curves  that  the 
SOR  Method  can  achieve  significantly  better  performance  than  the 
LMS  Gradient  Method  or  Jacobi  Relaxation  while  still  having  an 
average  number  of  operations/sample  which  is  on  the  order  of  the 
length  of  the  filter  N.  A comparison  of  the  "actual"  rates  of 
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convergence  of  the  three  algorithms  will  be  made  via  computer 
simulation  in  Chapter  VI. 
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5.6  Conclusion 

The  convergence  properties  of  the  LMS  Gradient  Algorithm, 
the  Jacobi  Relaxation  Method  and  the  SOR  Algorithm  have  been  ex- 
plored by  the  use  of  the  theoretical  bounds  derived  in  Chapters 
2 and  3.  The  dependence  of  the  rate  of  convergence  of  the  LMS 

I 

Gradient  Algorithm  on  the  value  of  the  parameter  u and  of  the 
rates  of  convergence  of  the  Fixed-Point  Iteration  Algorithms  on 
the  value  of  the  parameter  S was  demonstrated.  A decrease  in  the 
value  of  u in  the  LMS  Gradient  Algorithm  or  of  6 in  the  Jacobi 

I 

I 

Relaxation  Method  causes  a corresponding  increase  in  the  number 
of  sample  periods  required  for  the  convergence  cf  the  respective 
algorithm.  However,  such  a straightforward  relationship  between 
! the  value  of  8 and  the  rate  of  convergence  of  the  SOR  Method 
cannot  be  obtained  in  the  general  case. 

The  convergence  rates  of  the  Jacobi  Relaxation  Method  and 
the  SOR  Algorithm  when  the  algorithms  were  operating  under  the 
Variable  Iteration  Step  Method  were  also  considered.  As  expected,, 
a decrease  in  the  value  of  the  ratio  l/k  caused  a corresponding 
increase  in  the  number  of  sample  periods  required  for  convergence. 

| 

However,  the  Variable  Iteration  Step  Method  can  be  used  to  reduce  j 
the  average  number  of  operations/sample  required  by  the  two 

I 

fixed-point  iteration  algorithms  to  the  0(N)  required  by  the  LMS 
Gradient  Method.  The  total  average  number  of  operations/sample 


i 
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for  Adaptive  Fixed-Point  Iteration  is  still  slightly  higher  than 
that  required  by  the  LMS  Gradient  Algorithm  due  to  the  0(N) 
operations/sample  required  to  estimate  $xx  and  R^. 

A comparison  of  the  rates  of  convergence  of  the  LMS  Grad- 
ient Method  and  the  two  Fixed-Point  Iteration  Algorithms  with 


I 

| 


£/k  chosen  to  yield  0(N)  operations/sample  and  u chosen  to  equal- 
ize the  variance  of  the  estimates  of  the  statistics  revealed  that: 
the  SOR  Method  was  superior  to  both  the  LMS  Gradient  and  the 
Jacobi  Relaxation  Method.  This  suggests  that  the  SOR  Method  can 
offer  significantly  better  performance  than  the  LMS  Gradient 


Method  while  still  requiring  only  on  the  order  of  N operations 
per  sample.  This  hypothesis  will  be  tested  further  in  Chapter  VI 
where  the  "actual"  performance  of  the  three  iterative  algorithms 
will  be  compared  using  computer  simulation. 


I 


CHAPTT'R  VI  . 

INVESTIGATION  OF  PERFORMANCE  THROUGH  COMPUTER  SIMULATION 

In  the  previous  chapters,  the  well-known  LMS  Gradient 
Method  and  three  other  algorithms  for  use  in  adaptive  linear  es- 
timation have  been  discussed.  The  LMS  Gradient  Algorithm,  the 
Jacobi  Relaxation  Method  and  the  SOR  Algorithm  are  iterative  in 

1 

nature.  Therefore,  their  rates  of  convergence  are  of  extreme 
importance  in  an  evaluation  of  the  performance  of  these  algor- 
ithms. In  Chapter  V,  the  rates  of  convergence  of  these  three 
iterative  algorithms  were  studied  for  the  SKEP  problem  using  the 
theoretical  bounds  derived  in  Chapters  II  and  III.  The  theo- 
retical bounds  give  the  exact  rate  of  convergence  provided  that 
the  estimates  of  the  statistics  of  the  input  process  used  by 
the  adaptive  algorithm  are  unbiased  and  have  negligable  covar- 
iance (i.e.,  the  estimates  of  the  statistics  equal  their  exact 
values) . However,  in  practice  the  estimates  of  the  statistics 
of  the  input  process  used  by  an  adaptive  iterative  algorithm  have 
non-zero  covariance  and  ’’convergence"  of  the  algorithm  must  be 
interpreted  as  convergence  in  the  mean.  If  the  assumptions  made 
in  the  derivations  of  the  theoretical  bounds  in  Chapters  II  and 
III  are  reasonably  valid,  then  these  bounds  should  yield  a good 
approximation  to  the  convergence  in  the  mean  of  the  iterative 
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algorithms.  In  this  chapter,  the  convergence  properties  of  the 
LMS  Gradient  Method,  the  Jacobi  Relaxation  Algorithm  and  the  SOR 
Method  are  explored  via  computer  simulation.  The  results  of  the 

| simulation  are  compared  to  the  theoretical  bounds  derived  in 

j | 

Chapters  II  and  III  to  determine  how  well  these  bounds  predict 

the  performance  of  the  adaptive  algorithms  for  the  problem  con- 
sidered. 

In  contrast  to  the  three  algorithms  discussed  above, 
Levinson's  Algorithm  can  be  used  as  a non-iterative  approach  to 
adaptive  estimation.  While  Levinson's  Algorithm  is  more  complex 
than  the  three  iterative  algorithms  discussed  above,  it  deter- 
mines the  exact  solution  of  the  system: 

$XX  - = ^XX  . (6.1)  | 

I 

in  a single  step.  However,  if  Levinson's  Algorithm  is  to  be 
used,  $xx  must  be  constrained  to  be  a Topelitz  matrix  as  dis- 
cussed in  Chapter  IV.  The  "convergence"  of  the  adaptive  esti- 

I 

mation  procedure  using  Levinson's  Algorithm,  then,  depends  only 

A /A 

on  the  properties  of  the  estimates  4>xx  and  R^  and  upon  how 

often  Levinson's  Algorithm  is  applied  to  update  h . The  "con- 

~ 5-1 

vergence"  properties  of  the  adaptive  estimation  procedure  using 
. Levinson's  Algorithm  proposed  in  Section  4.2  are  also  considered 
in  this  chapter  using  computer  simulation. 

The  LMS  Gradient  Algorithm  requires  on  the  order  of  N 
operations/sample  for  its  implementation,  where  N is  the  filter 
length.  Unfortunately,  the  other  three  algorithms  considered 
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here  require  on  the  order  of  N operations/sample  when  implement- 
ed directly.  However,  -the  average  number  of  operations/sample 
can  be  reduced  for  the  two  Fixed-Point  Iteration  Algorithms  by 
the  use  of  the  Variable  Iteration  Step  Method  proposed  in 
Section  3.5.1.  A similar  approach  can  be  taken  for  the  adaptive 
estimation  method  using  Levinson's  Algorithm  as  discussed  in 
Section  4.2.  Therefore,  in  this  chapter  comparisons  of  the  rate 
of  convergence  of  the  four  adaptive  algorithms  will  be  made 
where  the  average  number  of  operations/sample  has  been  reduced 
to  roughly  0(N)  for  the  Jacobi  Relaxation  Method,  the  SOR  Al- 
gorithm and  Levinson's  Algorithm. 

The  computer  simulation  results  given  in  this  chapter 
allow  the  study  and  comparison  of  the  rates  of  convergence  of 
the  four  algorithms  considered  here  under  "practical"  conditions. 

That  is,  no  a priori  knowledge  of  the  signal  or  the  noise  is 

V 

supplied  to  any  of  the  algorithms.  Thus,  the  data  is  actually 
processed  adaptively  and  the  results  of  the  simulation  can  be 
used  to  verify  the  results  and  conclusions  given  in  Chapter  V 
which  were  based  on  the  theoretical  bounds  derived  in  Chapters 
II  and  III. 

I 

6.1  The  Problem  and  Simulation  Method 

In  Chapter  V,  the  convergence  properties  of  the  LMS 
Gradient  Algorithm,  the  Jacobi  Relaxation  Method  and  the  SOR 
Algorithm  were  investigated  for  the  signal-known-except-phase 
problem  using  the  theoretical  bounds  given  in  Chapters  II  and 


III.  To  verify  the  results  and  conclusions  of  Chapter  V,  the 
three  adaptive  algorithms  above  were  simulated  on  a digital 
computer  for  the  case  of  a single  sinusoidal  signal  of  fixed 
frequency  and  amplitude,  but  of  random  phase,  in  the  presence 
of  an  autoregressive  noise  sequence.  In  addition,  the  adaptive 
estimation  algorithm  using  Levinson's  Algorithm  proposed  in 
Section  4.2  was  also  simulated  as  well  as  the  optimal  filter  for 
the  SKEP  problem. 

The  single  sinusoidal  signal  sequence  used  in  the  simu- 
lation was  generated  as : 

S(n)  = A cos(ftn+0)  (5.1) 

where:  A = Signal  Amplitude 

Q = Signal  Frequency  (radians) 
n * Time  Index 

0 - Random  phase  distributed  uniformly  on  [-ir,ir]. 

The  data  sequence  was  then  formed  by  adding  an  autoregressive 
noise  sequence  to  the  signal  sequence.  An  autoregressive  noise 
sequence  was  generated  as  described  in  Section  5.1  as : 

"i-1  L 

N(n)  = l a Y(n-i)  (5.2) 

i=0 

where  Y(i)  are  independent  identically  distributed  random  var- 
iables with  zero  mean.  For  the  simulation  results  given  in 
this  chapter,  the  random  variables  Y(i)  were  drawn  from  a zero 
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mean,  unit  variance  Gaussian  distribution.  Therefore,  N(n)  is 
also  Gaussian  with  zero  mean  and  variance  which  can  be  deter- 
mined from  Equation  (5.18).  The  signal-to-noise  ratio  was 
varied  by  multiplying  N(n)  by  a constant  to  change  the  noise 
variance  of  the  resulting  sequence  according  to  the  following 
relation : 

VAR [C  N (n) ] = C2  VAR [N (n) ] . (6.2) 

The  resulting  data  sequence  given  by: 

X (n)  = S(n)  + C N (n)  (6.3) 

was  processed  by  the  four  adaptive  algorithms  considered  as  well 
as  by  the  optimal  filter  for  the  SKEP  problem. 

It  is  important  to  note  that  while  the  data  sequence 
generated  for  the  simulation  as  described  above  is  composed  of 
a single  sinusoidal  signal  sequence  of  fixed  amplitude  and 
frequency,  but  with  a random  phase,  added  to  am  autoregressive 
noise  sequence,  this  information  is  not  available  to  the  adap- 
tive algorithms.  That  is,  while  the  statistics  of  the  signal  and 
noise  are  known  to  an  observer,  the  processors  have  no  a priori 
knowledge  concerning  the  signal  or  the  noise.  Thus,  the  results 
in  this  chapter  must  be  interpreted  as  the  performance  of  a pro- 
cessor, which  is  given  no  a priori  knowledge  of  the  signal  or 
noise,  conditional  to  the  fact  that  the  data  generated  consists 
of  a single  sinusoid  of  fixed  frequency  and  amplitude  added  to 
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an  autoregressive  noise  sequence. 

6.1.1  The  Simulation  Method 

The  simulation  of  four  adaptive  algorithms  was  carried 
out  on  an  IBM  370/165  computer.  A source  listing  of  the  PL-I 
program  is  given  in  Appendix  D.  The  basic  simulation  method 
was  to  generate  a data  sequence  as  discussed  in  the  last  section 
and  to  allow  the  four  adaptive  algorithms  to  operate  on  this 
input  data.  The  instantaneous  squared  estimation  error  given  by: 

e (n)  = (S(n)-S(n))2  (6.4) 

was  computed  at  each  value  of  n for  each  of  the  four  algorithms.  • 
The  estimated  "mean-square  error"  was  computed  as  the  average  of 
the  instantaneous  error  sequence  e(n)  over  an  ensemble  of  100 
simulation  runs.  That  is,  the  MSE  was  estimated  as: 

1 100  - 2 

MSE  (n)  = Z (Si(n)-Si(n)  )*  . (6.5) 

A value  for  the  signal  phase  0 was  chosen  randomly  before  the 
start  of  each  of  the  100  runs.  The  "performance"  curves,  which 
are  plots  of  tne  estimated  mean-square  error  vs.  the  time  index 
n were  generated  in  the  manner  described  above.  Plots  of  time 
records  of  the  signal  estimates  and  plots  of  the  associated 
spectra  are  the  results  of  a single  run  and  are  not  averaged  over 
an  ensemble.  Thus,  the  simulation  results  presented  in  this 
chapter  must  be  viewed  in  light  of  the  fact  that  the  curves 


plotted  are  based  on  a finite  number  of  realizations  of  the 
estimation  process  and  therefore  have  non-zero  variance. 


6.2  Performance  of  the  LMS  Gradient  Algorithm  for  Various 
Values  of  the  Parameter  y 


A theoretical  bound  on  the  rate  of  convergence  of  the 
mean-square  error  produced  by  the  filter  determined  by  the  LMS 
Gradient  Algorithm  was  derived  in  Section  2.3.  The  rate  of 
convergence  of  the  LMS  Gradient  Algorithm  for  the  SKEP  problem 
was  explored  for  various  values  of  the  parameter  y in  Section 
5.2  under  several  different  SNR  conditions  using  this  theoreti- 
cal bound.  In  this  section,  the  convergence  of  the  MSE  for  the 

• 

LMS  Gradient  Algorithm  is  studied  via  computer  simulation.  The 
results  of  this  simulation  are  compared  to  the  theoretical  bound 
derived  in  Section  2.3.  The  simulation  was  carried  out  accord- 
ing to  the  method  discussed  in  Section  6.1.  The  parameter 
values  required  for  the  generation  of  the  signal  and  noise  se- 
quences were  chosen  to  be  identical  to  those  used  in  Chapter  V. 
Figure  5.1  gives  a summary  of  the  parameter  values  used  for 
generating  the  data  sequence  for  the  simulation.  It  is  again 
emphasized  that  while  the  data  generated  corresponds  to  the  SKEP 
problem,  the  LMS  Gradient  Algorithm  is  not  given  this  informa- 
tion. Thus,  the  results  of  the  computer  simulation  should  be 
interpreted  as  the  performance  of  the  LMS  Gradient  Algorithm, 
which  is  given  no  a priori  knowledge  of  the  signal  or  the  noise, 
conditional  to  the  fact  that  the  input  data  to  the  LMS  Gradient 
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Algorithm  is  composed  of  a single  sinusoid  of  fixed  amplitude 
and  frequency  added  to  an  autoregressive  noise  sequence. 

The  results  of  the  computer  simulation  of  the  adaptive 
estimation  procedure  using  the  LMS  Gradient  Algorithm  are  given 
in  the  form  of  plots  of  the  estimated  mean-square  error  as  a 
function  of  the  time  index  n in  Figures  6 . 1 through  6.4.  Each 
figure  corresponds  to  a different  value  of  input  signal-to-noise 
ratio  and  contains  four  separate  plots.  Each  of  the  four  plots 
within  a figure  shows  the  theoretical  bound  on  the  MSE  : (smooth 
curve) , as  well  as  the  estimated  MSE  obtained  from  the  computer 
simulation,  for  a different  value  of  the  parameter  y.  In  addi- 
tion, the  horizontal  base  line  on  each  of  the  plots  is  the  MSE 
which  would  result  from  the  use  of  the  optimal  filter  for  the 
SKEP  problem.  In  all  cas'.s,  the  vertical  axis  is  the  MSE  axis 
and  the  horizontal  axis  gives  the  number  of  sample  periods. 

The  signal-to-noise  ratios  and  the  values  of  y chosen  for  the 
simulation  are  identical  to  those  used  in  Chapter  V and  are  noted 
in  the  figures. 

It  can  be  seen  from  Figures  6.1  through  6.4  that  the 
theoretical  bound  derived  in  Chapter  II  is  close  to  the  actual 
performance  of  the  LMS  Gradient  Algorithm  for  the  problem  con- 
sidered here.  However,  it  should  be  noted  that  the  difference 
between  the  theoretical  bound  and  the  simulation  results  in- 
creases as  the  SNR  decreases.  This  is  due  to  the  fact  that  the 
variance  of  the  gradient  estimate  used  by  the  LMS  Gradient 
Algorithm  increases  as  the  SNR  decreases  in  this  case.  The 
primary  difference  between  the  simulation  results  and  the 
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Figure  6.1  Comparison  of  Actual  and  Theoretical  Performance 
of  the  LMS  Gradient  Algorithm  for  Various  Values 
of  the  Parameter  y.  SNR  = +7.0  db. 
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Figure  6.2  Comparison  of  Actual  and  Theoretical  Performance 
of  the  LMS  Gradient  Algorithm  for  Various  Values 
of  the  Parameter  y.  SNR  = -3.0  db. 
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Figure  6.3  Comparison  of  Actual  and  Theoretical  Performance 
of  the  LMS  Gradient  Algorithm  for  Various  Values 
of  the  Parameter  y.  SNR  = -13.0  db. 
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theoretical  bounds  is  that  for  the  smaller  SNR  conditions,  the 
theoretical  bound,  and  the  simulation  results  do  not  appear  to 
converge  to  the  same  value.  This  difference  is  often  referred 
to  as  the  "misadjustment  error"  in  the  literature  [7].  Aside 
from  the  effects  of  the  misddjustment  error,  the  theoretical 

I 

bound  derived  in  Section  2.3  is  a good  approximation  to  the 
"actual"  MSE(n)  produced  by  the  filter  determined  by  the  LMS 
Gradient  Algorithm  for  the  problem  considered. 

6.3  Comparison  of  the  Theoretical  Bound  and  Simulation  Results 
for  the  Jacobi  Relaxation  Method. 

Equation  (3.75)  derived  in  Section  3.3.2  gives  a theo- 
retical  bound  on  the  rate  of  convergence  of  the  MSE  produced  by 
the  filter  determined  by  the  Jacobi  Relaxation  Method.  Through 
the  use  of  this  bound,  the  rate  of  convergence  of  the  Jacobi 
Relaxation  Method  was  studied  for  various  values  of  the  relaxa- 
tion parameter  8 under  several  different  conditions  of  input 
SNR  in  Section  5.3.  In  addition,  the  convergence  of  the  Jacobi 
Relaxation  Method  operating  under  the  Variable  Iteration  Step 
procedure  was  studied  in  Section  5.3.2  for  various  values  of 
the  parameters  Z and  Jc.  In  this  section,  the  convergence  of 
the  MSE  for  the  filter  determined  by  the  Jacobi  Relaxation 
Algorithm  is  explored  through  the  use  of  computer  simu'ation. 

The  results  of  the  computer  simulation  are  compared  to  the 
theoretical  bound  derived  in  Section  3.3.2. 
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The  simulation  was  carried  out  according  to  the  method 
outlined  in  Section  6.1.  The  Jacobi  Relaxation  Method  was  im- 
plemented directly  using  Equation  (3.76)  from  Section  3.3.3. 

The  autocorrelation  matrix  and  the  autocorrelation  vector  were 
estimated  using  Equation  (3.5)  in  Section  3.1.  The  parameter 
values  required  for  the  generation  of  the  data  sequence  are 
identical  to  those  used  in  Chapter  V and  are  summarized  in 
Figure  5.1.  It  should  again  be  noted  that  the  results  of  the 
simulation  must  be  interpreted  as  the  performance  of  the  Jacobi 
Relaxation  Algorithm,  which  is  not  given  any  a priori  knowledge 
concerning  the  signal  or  the  noise,  conditional  to  the  fact  that 
the  input  data  to  the  algorithm  is  composed  of  a single  sinusoid 
of  fixed  amplitude  and  frequency  in  the  presence  of  an  additive 
autoregressive  noise  sequence. 

6.3.1  Performance  of  the  Jacobi  Relaxation  Method  for  Various 

Values  of  the  Relaxation  Parameter  3 

The  convergence  of  the  Jacobi  Relaxation  Method  as  a 
function  of  the  relaxation  parameter  S was  studied  in  Section 
5.3  for  various  SNR  conditions  through  the  use  of  the  theoretical 
bound  on  convergence  derived  in  Section  3.3.2.  To  determine  the 
usefulness  of  this  theoretical  bound  in  predicting  the  rate  of 
convergence  of  the  Jacobi  Relaxation  Method,  the  algorithm  was 
simulated  using  the  same  parameter  values  for  the  signal  and 
noise  as  used  in  Chapter  V.  Figure  5.1  gives  a summary  of  the 
parameter  values  used  in  generating  the  data  sequence  for  the 


simulation. 
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The  results  of  the  computer  simulation  of  the  adaptive 
estimation  method  using  the  Jacobi  Relaxation  Algorithm  with 
various  values  of  the  relaxation  parameter  8 are  given  in  Fig- 
ures 6.5  through  6.8.  Each  figure  is  for  a different  condition 
of  SNR  ratio  with  each  plot  in  a figure  corresponding  to  the 
estimated  mean-square  error  vs.  the  time  index  n for  a different 
value  of  the  relaxation  parameter  8 . In  addition,  the  theoreti- 
cal bound  (smooth  curve)  for  the  particular  value  of  8 is  also 
shown  in  each  plot  to  facilitate  comparisons.  The  horizontal 
base  line  in  each  plot  is  the  MSE  which  would  result  from  the 
use  of  the  optimal  filter  for  the  SKEP  problem.  In  all  cases, 
the  vertical  axis  is  the  MSE  axis  and  the  horizontal  axis  gives 
the  time  index  n.  The  signal-to-noise  ratios  and  the  values  of 
6 used  in  the  simulation  are  identical  to  those  used  in  Chapter 
V and  are  noted  in  the  figures. 

Note  from  Figures  6.5  through  6.8  that  the  "actual"  per- 
formance of  the  Jacobi  Relaxation  Method  is  close  to  the 
theoretical  bound  derived  in  Chapter  III  for  the  problem  con- 
sidered here.  However,  as  the  SNR  is  decreased,  the  difference 
between  the  theoretical  bound  and  the  simulation  results  in- 
creases. The  primary  difference  between  the  theoretical  bound 
and  the  simulation  results  for  the  lower  SNR  conditions  is  that 
the. "actual"  MSE  appears  to  converge  to  a higher  value  than  does 
the  theoretical  bound.  Since  the  variance  of  the  estimate  of 
the  autocorrelation  function  given  in  Section  3.1  decreases 
roughly  as  1/n,  the  variance  decreases  rapidly  initially,  but 
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slowly  for  larger  values  of  n.  Thus,  the  actual  convergence  of 

I 

the  Adaptive  Fixed-Point  Iteration  Method  using  Jacobi  Relaxation 
is  governed  initially  by  the  convergence  rate  of  Jacobi  Relaxa- 
tion. However,  as  n increases,  convergence  is  very  slow  due  to 
the  fact  that  the  variance  of  the  estimate  of  the  statistics 
decreases  slowly.  This  effect  is  more  noticeable  at  the  lower 
SNR  conditions  due  to  the  fact  that  the  variance  of  the  estimate 
of  the  autocorrelation  function  given  in  Section  3.1  increases 
as  the  SNR  decreases  in  this  case.  Aside  from  the  differences 
caused  by  the  effect  mentioned  above,  the  theoretical  bound 
derived  in  Section  3.3.2  is  a good  approximation  to  the  "actual" 
MSE(n)  produced  by  the  filter  determined  by  the  Jacobi  Relaxa- 
tion Method,  using  the  estimates  of  $>xx  and  R^  proposed  in 
Section  3.1,  for  the  problem  considered. 

6.3.2  Performance  of  Jacobi  Relaxation  Using  the  Variable- 

Iteration-Step  Method 

The  Variable-Iteration-Step  Method  was  proposed  in  Section 
3.5.1  as  a method  for  the  reduction  of  the  average  number  of 
operations/sample  for  the  Fixed-Point  Iteration  Algorithms. 

In  this  procedure,  the  Jacobi  Relaxation  Method  was  used  to 
update  2-  elements  of  the  unit-sample  response  vector  every  k 
sample  periods.  Thus,  if  1/ k < N,  the  average  number  of  opera- 
tions/sample can  be  reduced  to: 


H/k  (N+l)  < N (N+l)  . 


(5.20) 
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Figure  6.6  Comparison  of  Actual  and  Theoretical  Performance 
of  the  Jacobi  Relaxation  Algorithm  for  Various 
Values  of  the  Parameter  6.  SNR.  * -3.0  db. 
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However,  such  a reduction  in  the  average  number  of  operations/ 
sample  is  accompanied  by  a corresponding  increase  in  the  number 
of  sample  periods  required  for  convergence.  The  rate  of  conver- 
gence of  the  Jacobi  Relaxation  Method  under  the  Variable  Itera- 
tion Step  Method  was  studied  in  Section  5.3.2  using  the  theoreti- 
cal bound  derived  in  Chapter  III.  The  usefulness  of  this 
theoretical  bound  was  explored  by  a computer  simulation  of  the 
algorithm.  As  in  Section  5.3.2,  the  value  of  the  relaxation 
parameter  S was  chosen  to  be  8=0.14  as  determined  from  Equation 
(3.72).  The  value  of  l was  chosen  to  be  Jl=N=16  in  all  cases  and 
the  algorithm  was  simulated  for  k=16  and  k=32  as  was  the  case  in 
Section  5.3.2. 

The  results  of  the  computer  simulations  are  given  in 
Figures  6.9  through  6.12.  Each  figure  is  for  a different  SNR 
condition  with  the  plots  in  the  figures  giving  the  "actual" 

MSE(n)  for  the  Jacobi  Relaxation  Algorithm  using  the  Variable 
Iteration  Step  Method.  The  two  plots  in  each  figure  are  for  the 
cases  k=16  and  k-32  respectively.  To  facilitate  comparisons,  the 
theoretical  bound  for  the  particular  value  of  k is  also  included 
on  each  plot.  The  horizontal  base  line  in  each  plot  is  the  MSE 
which  would  result  from  the  use  of  the  optimal  filter  for  the 
SKEP  problem.  The  vertical  axis  is  the  MSE  axis  in  all  cases 
and  the  horizontal  axis  gives  the  number  of  sample  periods. 

The  SNR  conditions  and  k values  corresponding  to  each  plot  are 
noted  in  the  figures. 

As  can  be  seen  from  Figures  6.9  through  6.12,  the  "actual" 
performance  of  the  Jacobi  Relaxation  Algorithm  using  the  Variable 
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Iteration  Step  Method  is  close  to  that  predicted  using  the 
theoretical  bound.  Again,  however,  the  difference  between  the 
theoretical  bound  and  the  "actual"  MSE(n)  increases  as  the  SNR 
decreases.  This  is  again  due  to  the  fact  that  the  variance  of 
the  estimate  of  the  autocorrelation  function  given  in  Section  3.1 
increases  as  the  SNR  is  decreased  in  this  case.  Thus,  the  effect 
of  the  rate  at  which  the  variance  of  the  autocorrelation  func- 
tion estimate  decreases  is  more  noticeable  at  the  lower  SNR 
conditions  as  discussed  in  the  previous  section.  Aside  from  this 
effect,  which  causes  the  "actual"  MSE(n)  to  appear  to  converge 
to  a value  higher  than  that  predicted,  the  theoretical  bound 
gives  a good  approximation  to  the  "actual"  MSE(n)  produced  by 
the  filter  determined  by  the  Jacobi  Relaxation  Algorithm  using 
the  Variable  Iteration  Step  Method. 

6.4  Comparison  of  the  Theoretical  Bound  and  Simulation  Results 
for  the  Successive  Over-Relaxation  Method 

A theoretical  bound  on  the  convergence  of  the  mean-square 
error  produced  by  the  filter  determined  by  the  SOR  Method  was 
derived  in  Section  3.4.1  and  is  given  by  equations  (3.91)  and 
(3.92).  The  convergence  properties  of  the  SOR  Method  for  var- 
ious values  of  the  relaxation  parameter  6 were  investigated  for 
several  different  SNR  conditions  using  this  bound  in  Section 
5.4.1.  Also,  the  convergence  of  the  SOR  Algorithm  operating 
under  the  Variable  Iteration  Step  Method  was  studied  in  Section 
5.4.2  for  various  values  of  the  parameters  l and  k.  In  this 
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Figure  6.9  Comparison  of  Actual  and  Theoretical  Performance 
of  the  Jacobi  Relaxation  Algorithm  Using  the 
Variable-Iteration-Step  Method.  8=0.14. 

SNR  = +7.0  db. 
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Figure  6.10  Comparison  of  Actual  and  Theoretical  Performance 
of  the  Jacobi  P.elaxation  Algorithm  Using  the 
Variable-Iteration-Step  Method.  8=0.14. 

SNR  = -3.0  db. 
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section,  the  convergence  of  the  MSE  for  the  filter  determined 
by  the  SOR  Algorithm  is  explored  through  the  use  of  computer 
simulation.  The  results  of  the  computer  simulation  are  compared 
to  the  theoretical  bounds  derived  in  Section  3.4.1. 

The  method  outlined  in  Section  6.1  was  used  in  carrying 
out  the  simulation  of  the  SOR  Algorithm.  The  SOR  Method  was 
implemented  using  Equation  (3.85)  from  Section  3.4.2  and  the 
autocorrelation  matrix  and  vector  were  estimated  using  Equations 
(3.6)  and  (3.7)  from  Section  3.1.  The  data  sequence  used  in  the 
simulation  was  generated  using  parameter  values  identical  to 
those  used  in  Chapter  V,  which  are  summarized  in  Figure  5.1. 

As  in  the  case  of  the  computer  simulations  of  the  LMS  Gradient 
and  Jacobi  Relaxation  Algorithms,  the  results  of  this  simulation 
should  be  interpreted  as  the  performance  of  the  SOR  Method,  which 
has  no  a priori  knowledge  of  the  signal  or  the  noise,  conditional 
to  the  fact  that  the  input  data  to  the  algorithm  is  composed  of 
a single  sinusoid  of  fixed  frequency  and  amplitude  in  the  pre- 
sence of  an  additive  autoregressive  noise  sequence. 

6.4.1  Performance  of  the  SOR  Method  for  Various  Values  of  the 

Relaxation  Parameter  6 

The  theoretical  bound  derived  in  Section  3.4.1  was  used 
to  study  the  convergence  properties  of  the  SOR  Algorithm  for 
various  values  of  the  relaxation  parameter  S under  several  SNR 
conditions  in  Section  5.4.  The  SOR  Algorithm  was  simulated  us- 
ing the  same  parameter  values  for  the  signal  and  noise  as  used 
in  Chapter  V to  determine  the  usefulness  of  this  theoretical 
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bound  in  predicting  the  convergence  of  the  SOR  Algorithm. 

Figures  6.13  through  6.16  give  the  results  of  the  computer 
simulations  of  the  adaptive  estimation  method  using  the  SOR 
Algorithm  for  various  values  of  the  relaxation  parameter  6. 

Each  figure  is  for  a different  condition  of  SNR  with  each  plot 
in  a figure  corresponding  to  the  estimated  MSE  vs.  the  time 
index  n for  a different  value  of  the  relaxation  parameter  8 . 

In  addition,  the  theoretical  bound  corresponding  to  the  particu- 
lar  value  of  8 is  also  included  in  each  plot  to  facilitate  com- 
parisons. The  MSE  which  would  result  from  the  use  of  the  opti-  ' 
mal  filter  for  the  SKEP  problem  is  given  by  the  horizontal  base 
line  in  each  plot.  In  all  cases,  the  MSE  is  given  along  the 
vertical  axis  and  the  number  of  sample  periods  is  given  along  the 
horizontal  axis.  The  values  of  the  parameter  B and  the  SNR 
conditions  used  in  the  simulation  are  identical  to  those  used 
in  Chapter  V and  are  noted  in  the  figures. 

It  can  be  seen  from  Figures  6.13  through  6.16  that  the 
actual  performance  of  the  SOR  Algorithm  is  close  to  the  theoreti- 
cal bound  derived  in  Chanter  III  for  the  case  considered  here. 
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However,  it  appears  from  the  figures  that  the  "actual"  MSF(n) 
converges  to  a value  which  is  greater  than  that  predicted  by  the 
theoretical  bound.  This  situation  is  more  noticeable  at  the 

lower  values  of  SNR  and  is  due  to  the  effect  of  the  rate  at 

. \ * 

which  the  variance  of  the  estimate  of  the  autocorrelation  func- 
tion given  in  Section  3 . 1 . decreases  as  was  the  case  with  the 
Jacobi  Relaxation  Algorithm.  This  effect  is  more  noticeable 
in  the  curves  for  the  SOR  Algorithm  due  to  its  high  rate  of 
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Figure  6.14  Comparison  of  Actual  and  Theoretical  Perform 
ance  of  the  SOR  Algorithm  for  Various  Values 
of  the  Parameter  8.  SNR**-3.0  db 
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Figure  6.15  Comparison  of  Actual  and  Theoretical  Perform- 
ance of  the  SOR  Algorithm  for  Various  Values 
of  the  Parameter  8.  SNR=-13.0  db 
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Figure  6.16  Comparison  of  Actual  and  Theoretical  Perform 
ance  of  the  SOR  Algorithm  for  Various  Values 
of  the  Parameter  3.  S>7R=-23.0  db 
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convergence  (note  that  the  plots  only  cover  a range  of  0 _<  n _<  8)  . 
Aside  from  differences  due  to  this  effect,  the  theoretical  bound 

derived  in  Section  3.4.1  is  a good  approximation  to  the  "actual"  ; 
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MSE(n)  produced  by  the  filter  determined  by  the  SOR  Algorithm. 

6.4.2  Performance  of  the  SOR  Algorithm  Using  the  Variable 
Iteration  Step  Method 

I 

The  Variable-Iteration-Step  Method  was  proposed  in  Section \ 
3.5.1  as  a method  for  the  reduction  of  the  average  number  of 
operations/sample  required  by  the  fixed-point  iteration  algor- 
ithms. In  this  procedure,  the  SOR  Algorithm  is  used  .to  update 
£ elements  of  the  unit  sample  response  every  k samples.  There- 
fore, if  £/k . < N,  then  the  average  number  of  operations/sample 
can  be  reduced  to : 

£/k  (N+l)  . < N (N+l)  . (5.21) 

Such  a reduction  is,  obviously,  accompanied  by  a corresponding 
increase  in  the  number  of  sample  periods  required  for  convergence. 
The  performance  of  the  SOR  Algorithm  using  the  Variable-Itera- 
tion-Step  Method  was  investigated  in  Section  5.4.2  through  the 
use  of  the  theoretical  bound  given  by  Equations  (3.91)  and 
(3.92).  The  usefulness  of  this  theoretical  bound  was  studied 
by  the  use  of  a computer  simulation  of  the  algorithm.  For  the 
simulation,  the  value  of  the  relaxation  parameter  was  chosen  to 
be  6 = 1.0  and  the  algorithm  was  simulated  for  the  two  cases  of 
(£=1,  k»l)  and  (£=16,  k=16)  as  in  Section  5.4.2. 
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Figures  6.17  through  6.20  give  the  results  of  the  computer 
simulation  of  the  SOR  Algorithm  using  the  Variable-Iteration- 
Step  Method.  Each  figure  corresponds  to  a different  SNR  condi-  j 
tion  with  the  plots  in  each  figure  giving  the  "actual"  MSE(n) 
for  the  SOR  Algorithm  using  the  Variable-Iteration-Step  Method. 

The  two  plots  in  each  figure  are  for  the  cases  of  (£=1,  k=l) 
and  (Jt=16,  k=16)  respectively,  the  theoretical  bound  for  the 
particular  values  of  l and  k is  included  in  each  plot  to  facili- 
tate comparisons.  The  MSE  which  would  result  from  the  use  of 
the  optimal  filter  for  the  SKEP  problem  is  given  by  the  hori- 
zontal base  line  in  each  plot.  In  all  cases,  the  vertical  axis 
is  the  MSE  axis  and  the  horizontal  axis  gives  the  number  of  sample 
periods. 

Note  from  the  figures  that  the  "actual"  performance  of 
the  SOR  Algorithm  using  the  Variable-Iteration-Step  Method  is 
close  to  that  predicted  using  the  theoretical  bound.  Also  note 
that  the  difference  between  the  theoretical  bound  and  the 
"actual"  MSE(n)  increases  as  the  SNR  decreases  and  that  the  two 
curves  appear  to  "converge"  to  different  values.  These  differ- 
ences are  again  due  to  the  fact  that  the  variance  of  the  esti- 
mate of  the  autocorrelation  function  given  in  Section  3.1  in- 
creases as  the  SNR  decreases  in  this  case  making  the  effect  of 
the  rate  at  which  the  variance  decreases  more  noticeable  at 
the  lower  SNR  conditions  as  discussed  in  previous  sections. 

Aside  from  the  differences  caused  by  this  effect,  the  theoreti- 
cal bound  gives  a good  approximation  to  the  "actual"  MSE(N) 
produced  by  the  filter  determined  by  the  SOR  Algorithm  using  the 
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. 17  Comparison  of  Actual  and  Theoretical  Perform' 
ance  of  the  SOR  Algorithm  Using  the  Variable 
Iteration  Step  Method.  B=1.0.  SNR=+7.0  db. 
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Figure  6.18  Comparison  of  Actual  and  Theoretical  Perform 
ance  of  the  SOR  Algorithm  Using  the  Variable 
Iteration  Step  Method.  6=1.0.  SNR=-3.0  db. 
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Figure  6.19  Comparison  of  Actual  and  Theoretical  Perform' 
ance  of  the  SOP.  Algorithm  Using  the  Variable 
Iteration  Step  Method.  8=1.0.  SNR=-13.Q  db 
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Figure  6.20  Comparison  of  Actual  and  Theoretical  Perform- 
ance of  the  SOR  Algorithm  Using  the  Variable 
Iteration  Step  Method.  3=1  0.  SNR=-23.0  db 
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Variable-Iteration-Step  Method  for  the  problem  considered  here. 

i 

6 . 5 Comparison  of  Adaptive  Estimation  Algorithms  Through 
Computer  Simulation 

J 

In  the  preceeding  chapters  four  algorithms  for  adaptive 

| 

linear  estimation  have  been  considered.  The  convergence  proper- 
ties  of  the  LMS  Gradient  Algorithm,  the  Jacobi  Relaxation  Al- 
gorithm and  the  .SOR  Algorithm  have  been  studied  under  various 
conditions  via  computer  simulation  in  the  previous  section  of 
this  chapter.  In  this  section,  the  "actual"  convergence  proper- 
ties of  the  LMS  Gradient  Algorithm,  the  Jacobi  Relaxation 
Method,  the  SOR  Algorithm  and  Levinson's  Algorithm  will  be  com- 
pared.  The  "actual"  curves  of  MSE  vs.  the  time  index  n for  the 

I 

four  algorithms  were  generated  by  a computer  simulation  carried 

out  according  to  the  method  outlined  in  Section  6.1.  As  mention-' 

I 

ed  previously,  the  simulation  results  should  be  interpreted  as 
the  performance  of  adaptive  algorithms , which  have  no  a priori 
knowledge  of  the  signal  or  the  noise,  conditional  to  the  fact 
that  the  input  data  to  the  algorithms  is  composed  of  a single 
sinusoid  of  fixed  frequency  and  amplitude  added  to  an  auto- 

| 

regressive  noise  sequence. 

The  LMS  Gradient  Algorithm  requires  fewer  operations/ 
j sample  than  the  two  fixed-point  iteration  algorithms  or  Levin-  ! 

; son's  Algorithm.  Also,  the  estimate  of  the  input  statistics 
used  by  the  LMS  Gradient  Algorithm  has  a larger  variance  than 
the  estimate  proposed  in  Section  3.1  which  was  used  with  the 
other  three  algorithms.  Therefore,  any  meaningful  comparison  of 
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the  four  algorithms  should  attempt  to  equalize  the  number  of 
operations/sample  and  the  variance  of  the  estimate  of  the  sta- 
tistics across  the  four  algorithms.  This  problem  was  addressed 

I 

in  Section  5.5,  where  the  theoretical  bounds  derived  in  Chapters 
II  and  III  were  used  to  compare  the  performance  of  the  LMS 
Gradient  Algorithm,  the  Jacobi  Relaxation  Algorithm  and  the 
SOR  Algorithm.  In  that  section,  the  Variable-Iteration-Step 
Method  was  used  with  the  Jacobi  Relaxation  and  SOR  Algorithms 
to  approximately  equalize  the  average  number  of  operations/ 
sample.  An  attempt  to  equalize  the  variance  of  the  estimates  of 
the  statistics  of  the  input  process  was  made  by  selecting  the 
value  of  the  parameter  according  to  Equation  (5.27).  In  the 
computer  simulations  considered  here,  the  parameter  values 
chosen  for  the  LMS  Gradient  Algorithm,  the  Jacobi  Relaxation 
Algorithm  and  the  SOR  Algorithm  are  identical  to  those  used  in 
Section  5.5.  Levinson's  Algorithm  was  implemented  as  outlined 
in  Chapter  IV  and  was  applied  at  every  k=2N=32  sample  periods 
in  the  simulation  to  reduce  the  average  number  of  operations  per 
sample  to  0(N)  as  discussed  in  Section  4.2.  The  parameter  val- 
ues used  for  the  algorithms  for  each  SNR  condition  simulated  as 
well  as  the  total  average  number  of  operations/sample  for  each 
algorithm  are  summarized  in  Figure  6.21.  As  was  the  case  in  the 
computer  simulations  discussed  in  the  previous  sections  of  this 
chapter,  the  parameter  values  required  for  the  generation  of 
the  input  data  are  identical  to  those  summarized  in  Figure  5.1. 

Figures  6.22  through  6.33  give  the  results  of  the  computer! 
simulation  of  the  four  adaptive  estimation  algorithms  considered 
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here  for  various  SNR  conditions.  Figures  6.22,  6.25,  6.28  and 
6.31  give  plots  of  the  estimated  MSE  vs.  the  time  index  n for 
the  four  algorithms  for  SNR  conditions  of  +7.0  db,  -3.0  db, 

-13.0  db  and  -23.0  db  respectively.  These  curves  were  obtained 
by  taking  the  average  of  the  instantaneous  squared  error  se- 
quence given  by  Equation  (6.3)  over  an  ensemble  of  100  simulation 

I 

runs.  The  true  signal  sequence,  optimal  signal  estimate  and 
signal  estimates  produced  by  the  four  adaptive  algorithms  are 
shown  in  Figures  6.23,  6.26,  6.29  and  6.32.  These  estimates  are 

the  results  of  a single  simulation  run  and  are  not  averaged  over  ; 

! 

an  ensemble.  The  results  given  in  these  figures  for  SNR  condi- 

I 

tions  of  +7.0  db,  -3.0  db , -13.0  db  and  -23.0  db  respectively. 
Figures  6.24,  6.27,  6.30  and  6.33  give  the  magnitude  of  the 
Discrete  Fourier  Transforms  of  the  true  signal  sequence,  optimal 
signal  estimate  and  signal  estimates  produced  by  the  four 
adaptive  algorithms.  These  DFT's  are  1024  point  DFT's  of  the 
last  20  points  of  the  sequences  shown  in  Figures  6.23,  6.26, 

6.29  and  6.32.  These  20  points,  corresponding  to  one  period  of 
the  signal,  are  repeated  51  times,  giving  a 1020  point  sequence, 
before  the  1024  point  DFT  is  computed.  Since  the  last  four 
points  of  the  1024  point  DFT  are  equal  to  zero,  there  is  a very 
slight  windowing  effect  evident  in  the  plots  of  the  DFT's. 

Note  from  Figure  6.22  that  for  the  high  SNR  condition,  the 
rates  of  convergence  for  the  LMS  Gradient,  Jacobi  Relaxation  and 
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Algorithms  as  can  be  seen  in  Figures  6.25,  6.28  and  6.31.  Thus, 
j it  appears  taht  the  SOR  Algorithm  can  achieve  significantly 
better  performance  than  the  LMS  Gradient  or  Jacobi  Relaxation 
Algorithms  while  still  requiring  only  on  the  order  of  N opera- 
tions/sample. 

An  interesting  effect  can  be  noted  in  the  plots  of  MSE(n) 
for  the  Levinson  Algorithm  in  Figures  6.22,  6.25,  6.28  and  6.31. 

In  these  simulations,  Levinson's  Algorithm  is  used  to  update  the 
unit-sample  response  vector  h at  sample  times  n=31,  n=63  and 
n*95  based  on  estimates  of  the  autocorrelation  matrix  and  vector 
which  are  made  at  sample  times  n=0 , n=31  and  n=63  respectively  so 
that  the  computations  may  be  spread  out  over  k=2N=32  samples  to 
reduce  the  average  number  of  operations/sample.  "Note  from  the 

I 

j figures  that  the  mean-square  error  is  actually  increased  after 

i I 

I the  first  time  that  h is  updated.  This  is  due  to  the  fact  that 

the  estimates  of  the  autocorrelation  matrix  and  vector  are  very  ' 

I 

poor  at  sample  time  n=0 . The  MSE  is  then  dramatically  decreased 
after  the  second  time  that  h is  updated  due  to  the  fact  that  the  1 

A 

and  have 
sample  points.  Aside  from 

reduces  the  MSE  more  rapidly  than  the  LMS  Gradient  and  Jacobi 
Relaxation  Algorithms  at  the  lower  signal-to-noise  ratios. 

The  ability  of  the  adaptive  estimation  algorithms  to  per- 

j 

form  the  estimation  task  can  also  be  judged  by  considering 

i 

Figures  6.24,  6.27,  6.30  and  6.33  which  show  the  magnitude  of  the 
DFT  of  the  signal  estimates.  How  "good"  the  estimate  is  can  be 

I 

judged  by  comparing  the  amount  of  signal  component  in  the  esti- 

r a 


estimates 


improved  with  the  addition  of  32  more 

i 

this  effect,  the  Levinson  Algorithm 
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mate  to  the  amount  of  the  other  frequency  components.  As  can  be 
seen  from  the  figures,  the  SOR  and  Levinson  Algorithms  produce 
"better"  estimates  than  do  the  LMS  Gradient  and  Jacobi  Relation 
Methods.  However,  the  plots  of  the  magnitude  of  the  DFT  should 
not  be  weighted  heavily  in  the  evaluation  of  the  algorithms 
since  they  are  the  results  of  a single  simulation  run  and  there- 
fore have  a high  variance,  especially  at  the  lower  SNR  condi- 
tions. 

Thus,  it  appears  from  the  results  of  the  simulation  that 
the  SOR  and  Levinson  Algorithms  perform  significantly  better 
than  the  LMS  Gradient  or  Jacobi  Relaxation  Algorithms  for  the 
conditions  of  the  computer  simulation. 


6.6  Conclusion 

The  convergence  properties  of  the  four  adaptive  estima- 
tion algorithms  considered  here  have  been  explored  through  the 
use  of  computer  simulation.  The  results  of  the  simulation  have 
shown  that  the  theoretical  bounds  derived  in  Chapters  II  and  III 
give  good  approximations  to  the  actual  MSE(n)  produced  by  the 
filters  determined  by  the  LMS  Gradient,  Jacobi  Relaxation  and 
SOR  Algorithms.  However,  due  to  the  effect  of  the  increase  in 
the  variance  of  the  estimates  of  the  statistics  with  a decrease 
in  SNR,  the  MSE  produced  by  the  adaptive  algorithms  differs  from 
the  theoretical  bound  by  a larger  amount  at  the  lower  SNR  condi- 


tions considered. 
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It  is  also  evident  from  the  results  of  the  simulation 
that  the  SOR  Algorithm  and  Levinson's  Algorithm  can  give  sub- 
stantially better  performance  than  the  LMS  Gradient  Algorithm 
and  the  Jacobi  Relaxation  Method  when  the  Variable-Iteration- 
Step  Method  has  been  used  to  reduce  the  number  of  operations 
to  0(N)  and  the  parameter  y in  the  LMS  Gradient  Method  has  been 
chosen  to  equalize  the  variance  of  the  estimates  of  the  statis- 
tics. This  suggests  that  the  SOR  Algorithm  and  Levinson's 
Algorithm  can  be  used  to  perform  the  adaptive  estimation  task 
with  substantially  better  performance  than  the  LMS  Gradient 
Algorithm  with  only  a modest  increase  in  the  average  number  of 
operations/sample  required. 
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Figure  6.21  Parameter  Values  and  Average  Number  of  Opera- 
tions/Sample for  the  Four  Algorithms  as 
Implemented  in  the  Computer  Simulation. 
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Figure  6.22  Comparison  of  Actual  Performance  of  the  Four 
Adaptive  Algorithms.  SNR  = +7.0  db 
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Figure  6.25  Comparison  of  Actual  Performance  of  the  Four 
Adaptive  Algorithms.  SNP.  = -3.0  db 
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Figure  6.28  Comparison  of  Actual  Performance  of  the  Four 
Adaptive  Algorithms.  SNR  = -13.0  db 
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Figure  6.33  Comparison  of  the  DFT's  of  the  True  Signal 
Optimal  Signal  Estimate  (SKEP)  and  Signal 
Estimates  Produced  by  the  Four  Adaptive 
Algorithms.  SNR  = -23.0  db. 


CHAPTER  VII 

SUMMARY  AND  SUGGESTIONS  FOR  FUTURE  RESEARCH 

Four  adaptive  algorithms  which  attempt  to  approximate  the 

■ 

optimal  unit-sample  response  for  a linear,  causal,  FIR  digital 
filter  required  to  minimize  the  mean-square  error  of  estimation  j 
have  been  considered  when  applied  to  a spectral  line  enhance- 
ment  problem.  The  algorithms  are  adaptive  in  that  no  a priori 
knowledge  of  the  signal  or  noise  statistics  is  available  and  the 

I 

algorithms  must,  therefore,  estimate  these  statistics  from  the 
input  data  and  solve  a system  of  linear  equations  to  obtain  an 
approximation  to  the  optimal  unit-sample  response.  Historically,  ; 
two  approaches  to  this  problem  have  been  used.  One  approach, 
not  considered  here,  is  to  receive  a block  of  data,  estimate  the 
autocorrelation  matrix  and  vector  from  this  data  and  solve  the 
system  of  linear  equations  given  by: 

Stx  * * Rxx  ' (7-1) 

The  resulting  unit-sample  response  is  then  used  to  filter  the 

; 

block  of  received  data.  Due  to  the  delay  required  to  implement 
this  procedure,  it  is  usually  an  off-line  approach.  The  second 
approach  has  been  to  use  a real-time  adaptive  algorithm  such 
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as  the  LMS  Gradient  Algorithm.  The  LMS  Gradient  Method  and 
three  algorithms  which  lie  somewhere  between  the  two  historical 
methods  in  terms  of  computational  complexity  and  performance 
have  been  explored  in  this  dissertation. 

7 . 1 Summary 

The  LMS  Gradient  Method  is  a widely  used  adaptive  algor- 
ithm for  the  solution  of  the  system  in  Equation  (7.1).  Chapter 
II  outlines  the  LMS  Gradient  Algorithm,  which  is  based  on  the 
Method  of  Steepest  Descent  for  the  minimization  of  a function 
of  several  variables.  An  instantaneous  estimate  of  the  gradient 
of  the  MSE  is  used  as  an  estimate  of  the  statistics  of  the  input 
process.  A parameter  y present  in  the  LMS  Gradient  Algorithm 
must  be  chosen  to  be  smaller  than  a certain  bound,  which  depends 
upon  the  statistics  of  the  input  process,  in  order  for  the  al- 
gorithm to  converge.  A theoretical  bound  on  the  rate  of  con- 
vergence of  the  LMS  Gradient  Algorithm  was  developed  in  Chapter 
II  by  deriving  a bound  on  the  MSE  as  a function  of  the  time 
index  n.  This  bound  revealed  that  the  rate  of  convergence 
of  the  LMS  Gradient  Algorithm  was  dependent  on  the  parameter  y . 
Another  view  of  the  LMS  Gradient  Algorithm  was  given  in  Chapter 
II  as  a result  of  the  derivation,  at  least  formally,  of  another 
algorithm  based  on  an  attempt  to  approximately  satisfy  the 
orthogonality  principle  using  direct  estimates  of  the  expected 
value  terms  involved.  The  resulting  algorithm  was  very  similar 
to  the  LMS  Gradient  Method  except  that  the  constant  parameter  y 
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was  replaced  by  a data- dependent  term.  This  derivation  gives 

j 

additional  insight  into  the  operation  of  the  LMS  Gradient 
Algorithm. 

As  an  alternative  to  the  LMS  Gradient  Algorithm,  a family  | 
of  Adaptive-Fixed-Point  Iteration  Methods  was  proposed.  In  ! 

this  approach,  a direct  estimate  of  the  autocorrelation  matrix 
and  vector  is  made  and  continually  updated  so  that  the  linear 
system  of  equations  (7.1)  results.  This  system  is  solved  by 
applying  a single  step  of  a fixed-point  iterative  method  at 
each  sample  time  n.  The  estimates  of  the  autocorrelation  matrix 
and  vector  are  arbitrary  with  the  restriction  that  they  be  un- 
biased and  have  a reasonably  small  variance.  However,  a moving 
average  estimate  was  suggested  in  Section  3.1  for  the  estimation 
of  these  statistics. 

Two  oarticular  fixed-point  iteration  algorithms  were 

l 

considered  in  detail  in  Chapter  III.  The  Jacobi  Relaxation 
and  Successive  Over-Relaxation  Algorithms  were  shown  to  be  con- 
vergent in  the  mean  provided  that  certain  restrictions  were 
placed  on  the  value  of  the  relaxation  parameter  S present  in 
both  algorithms.  These  restrictions  are  independent  of  the 

i 

statistics  of  the  input  data.  Theoretical  bounds  on  the  MSE 
produced  by  the  filters  determined  by  each  of  these  two  algor- 
ithms were  derived  as  a function  of  the  time  index  n.  These 
bounds  reveal  that  the  rates  of  convergence  of  the  Jacobi  Relax-  | 
ation  and  SOR  Algorithms  depend  on  the  value  of  the  relaxation 
parameter  6 used  in  each  algorithm. 
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The  Jacobi  Relaxation  and  SOR  Algorithms  require  on  the 
2 

order  of  N operations/sample  if  implemented  directly  as  opposed 
to  the  0 (N)  operations/sample  required  by  the  LMS  Gradient  Al- 
gorithm. Since  such  computational  complexity  may  be  prohibitive 
in  real-time  applications,  methods  for  the  reduction  of  the 
average  number  of  operations/sample  were  considered  in  Chapter 
III.  In  the  Variable-Iteration-Step  Method,  a fixed-point  iter- 
ation algorithm  is  used  to  update  Z of  the  elements  of  the  unit- 
sample  response  vector  every  k sample  periods.  Thus,  if 
1/ k < N,  the  average  number  of  operations  can  be  reduced  to  on 
the  order  of: 


Z/k  (N)  . (7.2) 

- 

However,  such  a decrease  in  computational  complexity  brings  about 

I * 

a corresponding  increase  in  the  number  of  sample  periods  required 
for  convergence.  This  approach  is  extremely  attractive  in  that 
it  allows  the  time  available  between  samples  to  be  fully  utilized 
for  computation  since  the  number  of  operations/sample  can  be  ad- 
justed. In  addition  to  the  Variable-Iteration-Step  Method,  a 
procedure  for  carrying  out  the  Jacobi  Relaxation  Algorithm  using 

1 

the  Fast  Fourier  Transform  was  proposed  for  cases  in  which  the 
autocorrelation  matrix  estimate  exhibits  a Topelitz  structure  as 
does  the  estimate  proposed  in  Section  3.1.  This  method  can  lead 
to  substantial  savings  in  computation  time  for  larger  values  on 
N.  Thus,  the  Adaptive  Fixed-Point  Iteration  Algorithms  proposed 
in  Chapter  III  offer  a very  flexible  approach  to  the  solution  of 
the  adaptive  estimation  problem. 
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When  the  estimate  of  the  autocorrelation  matrix  exhibits 

a Topelitz  structure,  Levinson's  Algorithm  can  be  used  to  solve 

the  system  of  equations  given  by  (7.1)  in  a very  efficient 

manner.  Levinson's  Algorithm  was  outlined  in  Chapter  IV  and  is 

a non-iterative  algorithm.  Levinson's  Algorithm  requires  on  the 
2 

order  of  2N  operations  for  its  implementation.  Therefore,  to 
reduce  the  average  number  of  operations/sample  it  was  proposed 

/*  f 

that  the  estimates  <^x  and  be  updated  as  each  data  sample 
is  received  and  that  Levinson's  Algorithm  be  used  to  solve  the 
system  (7.1)  at  intervals  of  k sample  periods.  In  this  case, 
the  average  number  of  operations/sample  required  for  the  solution 
of  (7.1)  can  be  reduced  to  on  the  order  of: 


Thus,  Levinson's  Algorithm  is  an  attractive  approach  in  the 
special  case  where  the  estimated  autocorrelation  matrix  has  a 
Topelitz  structure. 

The  four  adaptive  algorithms  considered  here  are  summar- 
ized in  Figure  7.1.  As  can  be  seen  from  the  figure,  the  LMS 
Gradient  Algorithm  has  the  advantage  of  being  very  simple  and 
requiring  only  0(N)  operations/sample.  However,  the  other  three 
algorithms  have  the  advantage  of  allowing  a variable  number  of 
operations/sample  which,  while  greater  than  that  required  by  the 
LMS  Gradient  Method,  can  still  be  reduced  to  0(N).  Levinson's 
Algorithm,  which  is  the  most  complex  of  the  four,  has  the  ad- 
vantage  of  being  non-iterative  in  nature.  That  is,  the  "exact” 


23 


solution  of  (7.1)  is  found  in  a fixed  number  of  steps.  However, 
the  usefulness  of  Levinson's  Algorithm  is  limited  due  to  the  con- 
straint that  the  estimated  autocorrelation  matrix  have  a Tope- 
litz  structure.  In  addition,  the  Levinson  Algorithm  is  sequential 
in  nature  and  not  well  suited  to  parallel  processing  as  are  the 
Jacobi  Relaxation  and  SOR  Algorithms. 

The  performance  of  the  LMS  Gradient  Algorithm,  the  Jacobi 
Relaxation  Method  and  the  SOR  Algorithm  was  studied  in  Chapter 
V using  the  theoretical  bounds  derived  in  Chapters  II  and  III. 

The  dependence  of  the  rate  of  convergence  of  the  LMS  Gradient 
Algorithm  on  the  parameter  y and  of  the  Jacobi  Relaxation  and 
SOR  Algorithms  on  the  relaxation  parameter  8 was  demonstrated. 

A reduction  in  the  value  of  y for  the  LMS  Gradient  Algorithm 
or  in  the  value  of  8 for  the  Jacobi  Relaxation  Algorithm  lead 
to  a corresponding  increase  in  the  number  of  sample  periods 
required  for  convergence  of  the  respective  algorithm.  However, 

I 

such  a straightforward  relationship  between  the  value  of  8 and 
the  convergence  rate  of  the  SOR  algorithm  does  not  exist.  In 

| 

addition  to  the  effect  of  the  value  of  8 , it  was  demonstrated 
that  a decrease  in  the  value  of  the  ratio  l/k  led  to  a corres- 
ponding  increase  in  the  number  of  sample  periods  required  for  the 
convergence  of  the  Jacobi  Relaxation  and  SOR  Algorithms  when  the 
Variable-Iteration-Step  Method  was  used.  A comparison  of  the 
rates  of  convergence  of  the  three  iterative  algorithms  was  made 
using  the  theoretical  bounds  with  the  parameters  l and  k for  the 
Variable-Iteration-Step  Method  chosen  so  that  approximately  N 
operations/sample  were  required  by  the  two  fixed-point  iteration 
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algorithms.  In  addition,  the  value  of  the  parameter  u was  chosen 
according  to  Equation  (5.27)  in  an  attempt  to  equalize  the  var- 
iance in  the  estimates  of  the  statistics  of  the  input  process. 

The  results  of  this  comparison  indicated  that  for  a high  SNR 

I 

condition  of  +7.0  db,  the  three  algorithms  have  essentially  the 
same  rate  of  convergence.  However,  at  the  lower  SNR  conditions 
of  -3.0  db,  -13.0  db  and  -23.0  db,  the  SOR  Algorithm  exhibited 
a substantially  higher  rate  of  convergence  than  did  the  LMS 
Gradient  Algorithm  and  the  Jacobi  Relaxation  Method.  The  LMS 

I 

Gradient  and  Jacobi  Relaxation  Algorithms  converge  at  approxi- 
mately the  same  rate  with  the  rate  of  convergence  of  the  Jacobi 
Relaxation  Algorithm  being  slightly  higher.  Thus,  the  results 
of  the  comparisons  of  the  theoretical  bounds  indicate  that  the 
SOR  Algorithm  can  give  substantially  better  performance  than  the 

j 

LMS  Gradient  or  Jacobi  Relaxation  Method  while  still  requiring 
only  on  the  order  of  N operations/sample. 

The  performance  of  the  four  adaptive  algorithms  considered  | 
here  was  investigated  through  extensive  computer  simulation. 

The  results  of  this  simulation,  given  in  Chapter  VI,  indicate 
that  the  theoretical  bounds  on  the  MSE  as  a function  of  the 

] 

time  index  n derived  in  Chapters  II  and  III  give  good  approxima- 
tions to  the  actual  MSE(n)  produced  by  the  filters  determined  by 

I 

the  LMS  Gradient,  Jacobi  Relaxation  and  SOR  Algorithms.  However, ; 
the  difference  between  the  theoretical  bound  and  the  actual  per- 
formance of  the  algorithms  increase  as  the  SNR  decreases.  This 

i 

is  due  to  the  fact  that  the  variance  of  the  estimates  of  the 

I 

input  statistics  increases  as  the  SNR  decreases  in  this  case. 


i 
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Due  to  the  "mis-adjustment  error"  the  actual  MSE(n)  for  the 
LMS  Gradient  Algorithm  does  not  converge  to  the  value  predicted 
by  the  theoretical  bound.  Similarly,  due  to  the  effect  of  the 
rate  at  which  the  variance  of  the  estimate  of  the  statistics  used 
by  the  Jacobi  Relaxation  and  SOR  Algorithms  in  the  simulation 
decreases,  the  actual  MSE(n)  produced  by  the  filter  determined 
by  these  algorithms  appears  to  converge  to  a value  which  is 
slightly  greater  than  that  predicted  by  the  theoretical  bound. 

A comparison  of  the  results  of  the  computer  simulations 
of  the  four  adaptive  algorithms  was  presented  in  Chapter  VI. 

For  this  comparison,  the  number  of  operations/sample  was  chosen 
to  be  on  the  order  of  the  length  of  the  filter  for  the  Jacobi 
Relaxation,  SOR  and  Levinson  Algorithms.  In  addition  the  value 
of  the  parameter  y for  the  LMS  Gradient  Algorithm  was  chosen 
according  to  Equation  (5.27)  in  an  attempt  to  equalize  the 
variance  of  the  estimates  of  the  input  statistics  across  the 
four  algorithms.  The  results  of  this  comparison  indicate  that 
the  SOR  Algorithm  and  Levinson's  Algorithm  offer  substantially 
better  performance  than  the  LMS  Gradient  and  Jacobi  Relaxation 

i 

Methods  while  still  requiring  on  the  order  of  N operations/sample . 

From  the  results  of  the  computer  simulation  and  from  a 
consideration  of  the  theoretical  bounds  derived  in  Chapters  II 
and  III  it  is  evident  that  Adaptive  Fixed-Point  Iteration  using 
the  SOR  Algorithm  offers  an  extremely  attractive  solution  to  the 
problem  of  adaptive  estimation  for  Spectral  Line  Enhancement. 

The  SOR  Algorithm  using  the  Variable-Iteration-Step  Method  gives 
substantially  better  performance  than  the  LMS  Gradient  Method  or 
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the  Jacobi  Relaxation  Algorithm  while  still  requiring  on  the 
order  of  N operations/sample . In  addition,  the  variable  number 
! of  operations/sample  allowed  by  the  Variable-Iteration-Step 

Method  enables  the  processor  to  make  full  use  of  the  time  availa- 
ble between  samples.  Also,  the  Adaptive  Fixed-Point  Iteration 
Method  is  well  suited  to  parallel  processing. 

Levinson's  Algorithm  also  performed  well  in  the  simula- 
tion study.  However,  the  restriction  that  the  estimated  auto- 

} 

correlation  matrix  have  a Topelitz  structure  makes  it  less 
attractive  than  the  SOR  Method  as  a general  algorithm. 

Thus,  of  the  four  algorithms  considered.  Adaptive  Fixed- 

) 

Point  Iteration  using  the  SOR  Algorithm  and  the  Variable  Itera- 
tion Step  Method  appears  to  offer  the  most  attractive  solution 
to  the  problem  of  adaptive  linear  estimation  for  Spectral  Line 
Enhancement. 

I 

7.2  Suggestions  for  Future  Research 

Methods  for  the  solution  of  the  adaptive  linear  estima- 
tion  problem  have  been  proposed  in  this  thesis  which,  in  light 
of  the  results  given  here,  appear  to  yield  better  performance 
than  the  LMS  Gradient  Algorithm  without  a prohibitive  increase  in  j 
the  number  of  operations/sample  required  for  implementation. 

j 

The  performance  of  the  LMS  Gradient  Algorithm,  the  Jacobi 
Relaxation  Method,  the  SOR  Algorithm  and  Levinson's  Algorithm 
has  been  studied  for  a specific  set  of  input  conditions.  That 
is,  it  was  assumed  here  that  the  input  to  the  Spectral  Line 


Enhancer  is  in  the  form  of  a single  sinusoidal  signal  of  fixed 
amplitude  and  frequency,  but  of  random  phase,  added  to  an 
autoregressive  noise  sequence.  While  the  performance  of  the  LMS 

i 

Gradient  Algorithm  for  other  input  conditions  has  been  reported 
on  in  the  literature,  an  investigation  of  the  performance  of  the 
Adaptive  Fixed-Point  Iteration  Algorithms  and  Levinson's  Algor- 
ithm under  various  conditions  of  signal  and  noise  is  needed. 

Of  obvious  interest  are  the  problems  of  a signal  composed  of 
multiple  sinusoids,  signals  of  time-varying  frequency  and/or 
amplitude  and  signals  in  time-varying  noise  environments. 

In  general,  it  is  suggested  that  future  work  be  directed 
towards  a better  understanding  of  the  performance  of  the  algor- 
ithms proposed  here  and  include  evaluations  of  the  performance 
of  these  methods  as  compared  to  that  of  more  established  pro- 
cedures such  as  the  LMS  Gradient  Algorithm  for  a variety  of 
signal  and  noise  conditions. 
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APPENDIX  A 
PROOF  OF  THEOREM  2.1 

Theorem  2 . 1 

A necessary  and  sufficient  condition  for  the  convergence  of  the 
fixed-point  iteration  given  by: 

Y . = B Y + d (A. 1) 

— n+l  — n — 

is  that  the  magnitude  of  the  eigenvalues  of  the  iteration  mat- 
rix B be  less  than  1. 

Proof : 

By  Jordan's  Theorem,  there  exists  a non-singular  matrix  M and  a 
matrix  J of  Jordan  Form  such  that: 

B = M_1  J M . (A. 2) 


Thus : 


(A.  3) 


Since  M is  non-singular,  the  mapping: 


243 


Z -MY 
— n — n 


(A. 4) 


is  one-to-one  and  continuous.  Thus,  a necessary  and  sufficient 
condition  for  the  convergence  of  (A.l)  is  that: 


Z . = J Z + M d 

— n+1  — n — 


(A. 5) 


converges.  Consider  partitioning  (A. 5)  as  follows: 


Jx  0 0 

0 j2  . 


+ M d 


(A. 6) 


-k  n+1  0 0 ....  Jk  Zk 


where : 


J.  = A.  1 0 ...  0 

i l 

0 Ai  1 0 . . 0 
0 0 Ai  1 . . 0 


eigenvalue  of  the 
matrix  B. 


0 0 . . . . Ai 


Thus,  (A. 6)  can  be  expanded  into  K independent  equations: 
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Z.  ..  - J.  Z.  + (M  d)  . 
— i,n+l  1 — i,n  — 1 


(A.  7) 


where : 

(M  d) ^ = ith  partition  of  the  vector  (M  d) . 

Therefore,  a necessary  and  sufficient  condition  for  the  con- 
vergence of  (A. 5)  is  that  (A. 7)  converge  for  all  1 _ i _ K. 
Consider  the  last  equation  of  the  system  (A. 7)  given  by: 


zl,n+l  " U zH,n  + (M  4>t 


(A. 8) 


which  is  independent  of  the  other  (£-1)  equations.  Equation 
(A. 8)  can  be  rewritten  as: 

n+1  n . 

z£,n+l  " Xi  z£,0  + *mQ  Xi 


or: 


n+l  1+X? 

Z£, n+1  = Xl  z£ , 0 + T+X7  (M  -}£ 


(A. 9) 


Obviously,  (A. 9)  converges  if  |\^|  < 1.  Thus,  it  is  necessary 
that  | | . <1  for  (A.  8)  to  converge.  Now  consider  the  next  to 
the  last  equation  in  (A. 7)  given  by: 


zn  , = X.  z „ . + z.  + (M  d)  „ . 

£-l»n+l  l £-l,n  £,n  — £-1 


which  can  be  rewritten  as: 


= An+1 
1 
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zJl-l,n+l 


Za-1,0  + 


(l+xj) 

-ITX- 


t (A-10) 


It  is  obvious  that  a sufficient  condition  for  the  convergence  of 

(A. 10)  is  that  (A. 9)  converge  and  | X^|  <1.  It  can  be  seen  that 

a sufficient  condition  for  the  convergence  of  z.  . is  that 

£-],n 

z,  . , „ converge  and  that  |X.  | <1. 

^ j — -L  / n i 


Thus,  a necessary  and  sufficient  condition  for  the  convergence  of 
(A. 7)  is  that  | X^|  < 1.  Therefore,  it  is  necessary  and  suffi- 
cient that  . | X^  | < 1 for  all  1 _<  i _<  k to  insure  the  convergence 

of  (A. 5)  and,  thus,  of  (A. 2). 


APPENDIX  B 


APPROXIMATION  OF  MSE(n)  FOR  THE  LMS  GRADIENT  METHOD 


Equation  (2.31)  gives  an  approximation  to  the  mean-square 
error  produced  by  the  filter  determined  by  the  LMS  Gradient  Al- 
gorithm. This  appendix  contains  the  development  of  this  equa- 

I J 


tion.  The  true  MSE  produced  by  the  LMS  Gradient  Algorithm  can  be 
written  as : 


MSE  (n)  =E  [S2  (n)  ) -2E  [S  (n)  XT  (n-n^  1^]  +E  [h^X  (n-n^  XT  (n-n^  h^]  . (B.l) 
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i ! 


E[h.  X.  X.  h.] 
1 1 3 J 


(B.  4 ) 


Assume  that  h^,  h^,  and  X^  are  jointly  Gaussian.  Then,  (B.4) 
can  be  rewritten  as : 


E[h±  xi  h.  X.]  = E[hi  Xi]E[hj  X.]  + E[hi  X.]E[Xi  h.) 


+ E[hi  hj]E[Xi  Xj  ] 


(3.5) 


Now,  Widrow  [7]  has  shown  that  for  small  u and  low  input  SNR, 
h and  X exhibit  little  statistical  correlation.  Thus,  (B.5) 
can  be  rewritten  as : 


E[h±  X±  Xj  hj]  = E[hi]E[Xi]E[Xj]E[hj]  + E [hi  ] E [X  JE  [Xi  ] E [h  . ] 


+ E[hi  h.]E[Xi  X.] 


(B.  6) 


Consider  the  covariance  of  h^  and  h^  given  by: 


COV[hi,hj]  = E[(hi-hi)  (h.-hj)  ] = E[hi  h.I-h^. 


Now,  Widrow  [7]  has  shown  that  for  uncorrelated  input  data 
sequences : 


y E 

min 


COVth^hj]  = 


(B . 8 ) 


otherwise 


(B.  7) 


whe  re : 


Emin  = MSE  Produced  by  the  optimal  filter. 


Thus,  from  (B.7)  and  (B.8)  it  can  be  seen  that  if  u*E  . is 

min 

"small",  (B.6)  can  be  rewritten  as: 


E [h  . X.  X.  h.]  = E [h . ]E  [X.  ]E  [X  . ]E  [h  . ] + E [h  . ] E [X  . ] E [X . ] E [h  . ] 

iiJJ  iXJJ  J-JJ-J 

+ E[hi]E[hj]E[Xi  Xj]  . (B.6) 

Therefore,  under  the  assumptions  of  uncorrelated  input  data,  and 
small  y,  (B.2)  can  be  rewritten  as: 

E[hJ]E[X(n-n1  )XT(n-n. ) ]E[h  ] . (B.7) 

— n — 1 — 1 — n 


Now,  consider  the  term: 


N 


E [S  (n)  X (n-n.  ) h]  = E[S(n)  E X.  h . ] 
1 i=l  1 1 


or: 


T N 

E [S  (n)  X (n-n. ) h]  = E E[S(n)X.  h.  ] . 

i=l  11 


(B.8) 


Expanding  S (n)  X^  h^  into  a Taylor  series  about  (E [S (n) X^] ,E [h^] ) 
and  retaining  up  to  second  order  terms  yields: 
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E[S(n)Xi  h±]  = E[S(n)Xi]E[hi]  + 1/2  E [h±-E  [h±]  ] 

. * I 

I 

+ 1/2  E[S(n)Xi  - E[S(n)Xi]] 
or: 

il 

E[S(n)Xi  hi]  * E[S(n)Xi]E[hi]  (B.9) 

Thus,  from  (B.9)  it  can  be  seen  that  if  the  random  variables 
S(n)X^  and  h^  have  a probability  density  which  is  concentrated 
near  its  center  of  gravity,  then  (B.8)  can  be  rewritten  as: 

E[S(n)XT(n-n1)h]  = E (S  (n)  XT  (n-n^  ] E [h]  . (B.10) 

Combining  the  results  of  (B.7)  and  (B.10)  gives: 

MSE(n)  * E [S2  (n)  ] - 2E  (S  (n)  XT  (n-n^  ]E  [hJ+E  [h£]  E [1^]  . (2.31)1 


APPENDIX  C 


MEAN  AND  VARIANCE  OF  THE  TERM 

XT  (n-n^)  X (n-n1) 

X (n+l-n^) X (n+l-n^) 


In  this  Appendix,  a second  order  approximation  to  the 
mean  of  the  term: 

XT (n-n, ) X (n-n. ) 

_= ±-= ± (C.  1) 

X (n+l-nl)X(n+l-n1) 

is  derived  assuming  that  the  process  X(n)  is  stationary.  Con- 
sider the  sequence  of  r.v.  defined  as: 


Y. 

i 


X2 (n+l-n^-i) 


(C . 2 ) 


Equation  (C.l)  can  then  be  rewritten  as: 


f (Y) 


N-2 


i=-l 


Y. 

i 


N-l 
E Y. 
i=0  1 


= 1 + 


Y -Y 
-1  XN-1 

N-l 

E Y 

i=0  1 


(C.  3) 


Consider  expanding  the  above  expression  in  a Taylor  series  about 
the  point: 


where : 


I = IY-1  Y0  Y1 


Vi' 


Thus,  retaining  only  through  second  order  terms,  (C.3)  can  be 
rewritten  as : 


f (Y)  = f(E[Y])+Vf(Y-E[Y])+(Y-E[Y])TV2f(Y-E[Y])  (C.4) 

where : 

Vf  = gradient  of  f evaluated  at  E [Y] 

2 

V f = Hessian  Matrix  of  f evaluated  at  E [Y]  . 


.Consider  the  components  of  the  gradient  vector  Vf  evaluated  at 
the  point  E [Y] ; recalling  that  X was  assumed  stationary: 


The  elements  of  the  Hessian  Matrix  V f evaluated  at  the  point 
E[Y]  are  as  follows: 


H 1 

22f 

. 2iY-rV 

= 0 

E [Y] 

. 

t , 

2Y.2Y . 
i 3 

E [Y] 

N-l  - 

•E  Y 

K=0  K 

22f 

-1 

-1 

2Yi2Y_1 

ELY] 

N-l  , 

z yk 

K=0  K 

E(Y1  Y E ty] 

i / j^l 

i , j=N-l 


(C . 8 ) 


i^-1  (C. 9) 


N-l 


N-l 


22  f 


2Yi2Vi 


1 -2(Vl"Y-l)  E yk 

K=0  K=0 


E[Y] 


N-l 

E YI 
K=0  * 


E[Y] 


i^-1 

i^N-l 


22f 


?* 


-1 


~T~2 

U E*  [Y] 


(C.10) 


(C.ll) 


E [ Y] 


22f 


~7 

2ZY 


N-l 


-1 


N-l 

E [ Y]  E Y 

j=0  J 


-1 

TjT 


(C.12) 


2 N E [Y] 


From  Equations  (C.5)  through  (C.12)  it  can  be  seen  that  (C.4) 


can  be  written  as: 


f (Y)  = f (E  [ Y]  ) + 


N E [Y] 


2(yn-i'y-i}  n'2 

+ ~ j>  — 2 (Y  -E[Y.  ]) 

1TE^[Y]  i=0  1 1 


(VfE[Vi1)(Vi+Y-f2Em) 


2 2 

N E^[Y] 


(Y  ,-Y„  .)  2 (Y_, -Y  ) N-2 

f(Y)  : f (E  [Y]  ) + 2 i-  S (Y.-E[Y.  ]) 

N ElYj  N^E^tY]  i=0  1 1 


(VrE[YN-i]) 


N E [Y] 


(Vi+Y-r2E[Y1) 


(C.  13) 


Thus,  the  mean  of  f(Y)  can  be  written  as: 


E[f (Y) ] 


“ -H Z (E[Y-iYi^EtYN-lYi]) 

N E [Y]  i=0  11 

N2  E2[Y] 


(C . 14 ) 


or  if  X is  assumed  Gaussian: 


E[f  (Y)  ] 


1 5-5 Z (R*  (i-l)-RjY(i-N+l)) 

NZR^X(0)  i-0  ** 


3Rxx(0)+2Rxx(N_1'i) 

n2  4c<°> 
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Since  R^fK)  has  a maximum  at  K=0 : 

|E[f(Y)]-l|  . <2i2^iL  + . (C.15) 

- N N 

Thus,  it  can  be  seen  from  (C.15)  that  E[f(Y)]-l  for  large  values 
of  N. 

Determination  of  the  variance  of  f (Y)  is  difficult.  How- 
ever, intuitively,  it  can  be  argued  that  the  variance  becomes 
small  as  N becomes  large.  From  Equation  (C.3),  it  can  be  seen 
that  f ( Y)  can  also  be  written  as: 


f (Y) 


1 

N 

1 

N 


N-2 

Z 

i=-l 

N-l 


Yi 


Z 

i=0 


Y . 
l 


(C. 16 ) 


or,  in  terms  of  the  sample  autocorrelation  function  R^,  dis- 
cussed in  Section  3.1: 


f (Y) 


RXX ( 0 )_Ntn 


n+1 


RXX (0)N, 


(C. 17) 


Now,  it  was  shown  in  Section  3.1  that  the  variance  of  Rxx  ap- 
proaches zero  as  N + ®,  Thus,  both  the  numerator  and  denominator 
in  (C. 17)  approach  constants  with  probability  1 as  N approaches 
<».  Thus,  intuitively,  the  variance  of  f(Y)  becomes  small  as 


N becomes  large. 


APPENDIX  D 

COMPUTER  PROGRAMS  FOR  THE  STUDY  OF  FOUR  ADAPTIVE 
LINEAR  ESTIMATION  ALGORITHMS 

This  appendix  describes  the  computer  programs  used  in  the 
study  of  the  LMS  Gradient,  Jacobi  Relaxation,  SOR  and  Levinson 
Algorithms.  Four  separate  programs  were  used  in  the  study.  One 
program  was  used  for  the  simulation  of  the  four  adaptive  algor- 
ithms; two  were  used  in  computing  the  theoretical  bounds  derived 
in  Chapters  II  and  III  and  for  the-  comparison  of  these  bounds 
with  the  simulation  results,  and  the  fourth  program  was  used  to 
compute  the  Discrete  Fourier  Transform  of  data  generated  by  the 
simulation  program. 

I 

D.l  Simulation  Program 

The  four  adaptive  algorithms  as  well  as  the  optimal  filter 
for  the  SKEP  problem  were  simulated  on  an  IBM  370/165  computer. 

The  basic  simulation  method  was  to  generate  a signal-plus-noise 
data  sequence  and  to  process  this  data  sequence  using  each  of 
the  four  algorithms  as  well  as  the  optimal  filter.  The  instant- 
aneous squared  error  given  by  equation  (6.4)  was  computed  for 
each  of  the  five  filters  implemented.  The  mean-square  error 

was  estimated  as  the  ensemble  average  of  the  instantaneous  error 
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over  an  ensemble  of  100  signal  estimate  records. 

D.1.1  Specifications  of  the  Problem 

The  specifications  of  the  problem  are  discussed  in  Section 
6.1  and  are  summarized  below: 

I 

1.  Single  frequency  sinusoidal  signal  with  random  phase  j 
uniformly  distributed  on  [ — tt  , tt  ] specified  by  equation 
(5.1) . 

2.  Zero-mean  Gaussian  autoregressive  noise  sequence  with 

i 

autocorrelation  function  given  by  equation  (5.18). 

3.  Filter  length  N = 16. 

4.  Parameter  values  for  data  generation  summarized  in 
Figure  5.1. 

D.1.2  Program  Structure 

! 

The  structure  of  the  simulation  program  is  illustrated  in 
Figure  D.l.  Each  program  segment  is  described  below: 

j 

1.  ESTIM8  — Main  program  of  the  simulation,  contains  segments! 

ALLOC,  SIMULATE  and  OUTPUT. 

2.  ALLOC  — Reads  data  used  in  controlling  the  simulation  and 

parameter  values  for  simulation,  allocates  re- 
quired storage,  initializes  storage  locations. 

j 
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INPUTS: 

N — 

REL-LEN  — 
#REC  — 
DELTA  — 
ALPHA  — 

SNR  — 

LJAC  — 


K JAC  — 


LSOR  — 

KSOR  — 

KLEV  — 

BJAC  — 

BSOR  — 

MU  — 

H-INIT(N)  — 


I 

Filter  length. 

Number  of  samples/record. 

Number  of  records  to  be  processed. 
Delay  value  n^. 

Autoregression  parameter  of  noise 
sequence. 

Signal-to-noise  ratio. 
Variable-Iteration-Step  parameter 
l for  Jacobi  Relaxation  Method. 
Variable-Iteration-Step  parameter  ! 
k for  Jacobi  Relaxation  Method. 
Variable-Iteration-Step  parameter 
l for  SOR  Method. 

Variable-Iteration-Step  parameter 
k for  SOR  Algorithm. 

Parameter  k for  Levinson's  Al- 
gorithm. 

Parameter  B for  Jacobi  Relaxation 
Algorithm. 

Par^eter  3 for  SOR  Algorithm. 
Parameter  y for  LMS  Gradient  Al- 
gorithm. 

Initial  unit  sample  response 
vector . 


AMP 


Amplitude  of  the  signal. 


FREQ  — 
H-OPT(N)  — 


OUTPUTS : 
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Frequency  of  the  signal  (radians). 
Optimal  unit  sample  response 
vector. 


Same  as  Inputs. 

3.  SIMULATE  — Simulation  segment.  Produces  an  ensemble  of 

#REC  signal  estimate  records.  Contains  seg- 
ments INIT  and  RECORD. 


INPUTS: 

Same  as  ALLOC. 

OUTPUTS : 


SIGNUL(n)  — 


SEOPT (n)  — 


SELMS(n) 


SEJAC(n)  — 


SESOR(n)  — 


SELEV(n)  — 


Last  signal  record  computed  in 
the  simulation. 

Last  signal  estimate  record 
produced  by  the  Optimal  Filter 
in  the  Simula  cion. 

Last  signal  estimate  record  pro- 
duced by  the  LMS  Gradient  Algor- 
ithm in  the  simulation. 

Last  signal  estimate  record  pro- 
duced by  the  Jacobi  Relaxation 
Algorithm  in  the  simulation. 

Last  signal  estimate  record  pro- 
duced by  the  SOR  Algorithm  in  the 
simulation. 

Last  signal  estimate  record  pro- 
duced by  the  Levinson  Algorithm  in 


| 


the  simulation. 


l 


26 


MSEOPT(n)  — Estimated  MSE  produced  by  the 

optimal  filter. 

MSELMS(n)  — Estimated  MSE  produced  by  the  LMS 

Gradient  Method. 

MSEJAC(n)  — Estimated  MSE  produced  by  the 

Jacobi  Relaxation  Algorithm. 
MSESOR(n)  — Estimated  MSE  produced  by  the  SOR 

Algorithm. 

MSELEV(n)  — Estimated  MSE  produced  by  the 

Levinson  Algorithm. 

OUTPUT  — Outputs  the  results  of  the  simulation. 

INPUTS : 

Same  as  outputs  of  SIMULATE. 

OUTPUTS : 

Same  as  Inputs . 

5.  INIT  — Initializes  variables  prior  to  the  generation  of 

I 

each  record  in  the  simulation.  Calls  Procedure 

I 

RAN. 

OUTPUTS : 

PHASE  — Phase  of  the  sinusoid. 

In  addition  all  unit  sample  response  vectors  and 
autocorrelation  estimates  are  initialized. 

6.  RECORD  — Produces  a single  signal  estimate  record  for  each 

of  the  four  adaptive  algorithms  as  well  as  for  the  j 
optimal  filter.  Contains  segments  DATA,  SIG-EST, 


UPDATE  and  MSE. 


INPUTS : 


PHASE  — Phase  of  the  sinusoid. 

Initialized  unit  sample  response  vectors  and  auto 
correlation  estimates. 

OUTPUTS : 

Signal  estimate  records  for  each  of  the  five  fil- 
ters. Estimates  of  MSE(n)  for  each  of  the  five 
filters  based  on  all  the  data  previously  pro- 
cessed. 

RAN(IY)  — Produces  Uniform  [0,1]  random  numbers  using 
the  power  residue  method. 

INPUTS: 

IY  — Seed  number,  changed  by  RAN. 

OUTPUTS : 

RAN (IY)  — Uniform  [0,1]  number. 

DATA  — Generates  data  sample,  computes  autocorrelation 
estimate.  Contains  segment  AUTO-CORR.  Calls 
Procedure  XNOISE. 


INPUTS : 

N — 

AMP  — 
FREQ  — 
PHASE  — 
DELTA  — 
ALPHA  — 


Filter  length. 

Signal  amplitude. 

Signal  frequency  (radians) . 
Signal  phase  (radians) . 

Delay  value  n^. 
Autoregression  parameter  for 
noise  sequence . 
Signal-to-noise  ratio. 


SNR 


OUTPUTS 


9.  SIG-EST  — 


INPUTS : 


#SAMP  -- 
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Time  index. 


Signal  sample  at  time  n. 

Data  sample  at  time  n.  The  array 
X operates  as  a stack. 

Estimate  for  Topelitz  autocorre- 
lation matrix. 

Estimate  for  autocorrelation 
vector. 

Generates  signal  estimate  samples  for  each  of 
the  five  filters  simulated. 


SIGNUL(n)  — 
X (n)  — 

RXX(k)  — 

AUTO ( k ) — 


N — 

X(n)  — 
H-OPT  — 

H-LMS  — 

H-JAC  — 

H-SOR  — 

H-LEV  — 

SEOPT  — 
SEJAC  — 


Filter  length. 

Data  vector. 

Optimum  unit  sample  response 
vector. 

Unit  sample  response  for  LMS 
Gradient  Algorithm. 

Unit  sample  response  for  Jacobi 
Relaxation. 

Unit  sample  response  for  SOR 
Algorithm. 

Unit  sample  response  for  Levin- 
son's Algorithm. 

Optimal  signal  estimate. 

Jacobi  Relaxation  signal  esti- 


mate. 
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SELMS  — LMS  Gradient  Signal  estimate. 

SESOR  — SOR  signal  estimate. 

SELEV  — Levinson  signal  estimate. 

10.  UPDATE  — Updates  the  unit  sample  response  vectors  for 

the  four  adaptive  algorithms. 


INPUTS : 


N — 
H-LMS 


H-JAC 


H-SOR 


H-LEV 


OUTPUTS : 


H-LMS 


H-JAC 


H-SOR 


Filter  length. 

Unit  sample  response  vector  for 
LMS  Gradient  Algorithm  at  time  n. 
Unit  sample  response  vector  for 
Jacobi  Relaxation  Algorithm  at 
time  n. 

Unit  sample  response  vector  for 
SOR  Algorithm  at  time  n. 

Unit  sample  response  vector  for 
Levinson's  Algorithm  at  time  n. 

Unit  sample  response  vector  for 
LMS  Gradient  Algorithm  at  time 
n+1. 

Unit  sample  response  vector  for 
Jacobi  Relaxation  Algorithm  at 
time  n+1. 

Unit  sample  response  vector  for 
SOR  Algorithm  at  time  n+1. 

Unit  sample  response  vector  for 
Levinson's  Algorithm  at  time  n+1. 


H-LEV 


11. 
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MSE  — Computes  estimated  mean-square  error  based  on 
all  data  records  previously  processed. 


INPUTS : 

SIGNUL(n)  — 
SEOPT (n ) — 
SELMS(n)  — 

SEJAC(n)  — 

SESOR (n)  — 

SELEV(n)  — 

OUTPUTS : 

MSEOPT(n)  — 

MSELMS(n)  — 

MSEJAC(n)  — 

MSESOR(n)  ~ 

MSELEV(n)  — 


Signal  sample  at  time  n. 

Optimal  signal  estimate  at  time  n 
LMS  Gradient  Signal  estimate  at 
time  n. 

Jacobi  Relaxation  signal  estimate 
at  time  n. 

SOR  Algorithm  signal  estimate  at 
time  n. 

Levinson's  Algorithm  signal 
estimate  at  time  n. 

Estimated  MSE  for  optimal  filter 
at  time  n. 

Estimated  MSE  for  LMS  Gradient 
Algorithm  at  time  n. 

Estimated  MSE  for  Jacobi  Relaxa- 
tion at  time  n. 

Estimated  MSE  for  SOR  Algorithm 
at  time  n. 

Estimated  MSE  for  Levinson's 


Algorithm  at  time  n. 
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12. 

XNOISE  — Recursively 

generates  autoregressive  noise 

sequence.  Calls  Procedure  GAUSS. 

I 

INPUTS : 

ALPHA  — 

Autoregression  parameter  for 

noise  sequence. 

DELTA  — 

Delay  value  n^ 

#SAMP  — 

Time  index. 

Y — 

Vector  containing  previous  DELTA 

Gaussian  random  numbers. 

IY  — 

SEED  number. 

OUTPUTS : 

XNOISE  — 

Autoregressive  noise  sequence 

sample . 

Y — 

Previous  DELTA  Gaussian  random 

numbers . 

IY  — 

New  Seed  number. 

13. 

AUTO-CORR  — Computes 

autocorrelation  estimates. 

INPUTS : 

X (n)  -- 

Data  vector. 

N — 

Filter  length. 

OUTPUTS : 

RXX(k)  — 

Topelitz  autocorrelation  matrix 

estimate. 

AUTO (k)  — 

Autocorrelation  vector  estimate. 
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rHlS  PAGE  IS  BEST 

fftOM  OOP*  f UKBISHSD  TO  ODC 

D.1.3  Source  Program 


ADAPTIVE  LINE  ENHANCER  SIMULATION  6/13/7* 


THIS  PROGRAM  SIMULATES  A SPECTRAL  LINE  ENHA 
GRADIENT  PROCEDURE  PROPOSED  BY  wIDROw  ET.AL. 
1975).  IN  ADDITION*  ALGORITHMS  WHICH  ESTIma 
CORRELATION  MATRIX  AND  SOLVE  FOR  ThE  FILTER 
USING  THE  JACOBI  RELAXATION.  SOR  OR  LEVINSON 
SIMULATED  AS  WELL  AS  TnE  OPTIMUM  FILTER  FOR 
THE  INPUT  TO  ALL  CF  THE  SIMULATED  PROCESSORS 
SINUSOID  (WITH  USER  SPECIFIED  FREQ  AND  AMPLI 
i AUTOREGRESSIVE  NOISE  SEQUENCE.  THE  SIGNAL  A 
TO  BE  UNCORRELATED  AND  NOISE  SAMPLES  A DISTA 
APART  ARE  ALSO  ASSUMED  UNCORREL A TED . NOISE 
BY  PROCEDURE  XNOISE  ANO  ARE  ASSUMED  TO  HAVE 
WELL  AS  THE  CORRELATION  PROPERTIES  INDICATED 


■»•»**■*■»*■»*#*#■*■»*■»*»*** 

NCER  USING  THE  LMS 
(IEEE  PROC.  DEC. 

TE  THE  INPUT  AUTO- 
IMPULSE  RESPONSE 
ALGORITHMS  AKt  ALSO 
FOR  THf  SKER  PROBLEM. 

CONSISTS  OF  A SINGLE 
TUDE)  AOOED  TO  AN 
NO  NOISE  ARE  ASSUMED 
NCE  DELTA  OR  GREATER 
SAMPLES  ARE  GENERATED 
MEAN=0  AND  VAR=I  AS 
ABOVE. 


INPUT  DATA  » ***#•**»•»•»*•*«••»**»■»*•«■*****  * 

THF  FIRST  INPUT  RECORD  IS  IN  A DATA  FORM  AND  SPECIFIES  THE 
FOLLOWING  PARAMETERS: 

N = # OF  WEIGHTS  FOR  FOR  ALL  ALGORITHMS  (INTEGER) 

RECLEN  = RECORC  LENGTH  FOR  SIMULATION  RUN  (INTEGER) 

»REE  = * OF  REC  LEN  RECORDS  TO  BE  AVERAGED  (INTEGER) 

DELTA  = p SAMPLES  FOR  DELAY— SEE  ABOVE  (INTEGER) 

SNR  - DESIRED  S I GNAL-TO-NO ISE  RATIO  (REAL) 

MEAN  - DESIRED  MEAN  OF  THE  NOISE  (REAL) 

ALPHA  = AUTOREGRESSION  PARAMETER  (REAL)  (SEE  COMMENTS  IN  XNOISE5 


Lo  AC 
K J AC 
LSOR 
KSOR 
KLEV 
8S0R 
BJAC 


VARIA8LE  ITERATION  STEP  PARAMETER  L 
VARIABLE  ITERATION  STEP  PARAMETER  K 
VARIABLE  ITERATION  STEP  PARAMETER  L 
VARIABLE  ITERATION  STEP  PARAMETER  K 
p OF  SAMPLES  BETWEEN  APPLICATION  OE 

parameter  beta  for  the  sor  algorithm 


= parameter  beta 


E COMMENTS  IN  XNOISE) 
FOR  JACOBI  METHOD (INT) 
FDR  JACOBI  METHOD (INT) 
FOR  SOR  METHOD  (INT) 
FOR  SOR  METHOD  (INT) 
LEVINSON  mEThOO  (INT) 
(REAL) 


THE  JACOBI  RELAXATION  METHOD 


THE  NEXT  SET  OF  INPUT  DATA  IS  IN  A LIST  FORM.  THE  FOLLOWING 
DATA  IS  READ: 


AMP  = AMPLITUDE  OF  SIGNAL  » RE  ALT) 

FREQ  = FREQUENCY  OF  SIGNAL  IN  RADIANS  (REAL' 
IY  = SEED  NUMBER  FOR  RANDOM  NUMBER  GENERATION 


THE  LAST  INPUT  RECORD  GIVES  PARAMETERS  USEO  IN  THE 
ALGORITHMS.  A LI  S'!-  FORM  IS  USED  TO  ENTER: 

MU  = VALUE  OF  MU  TO  EE  USED  IN  LMS  PROCEDURE  (REAL) 
W_INIT(N)  = INITIAL  VALUES  OF  THE  M FILTER  WEIGHTS 
H OPT  s OPTIMAL  UNIT  SAMPLE  RESPONSE  (REAL) 


(OOO  INTEGER) 
the  Each  of 


(REAL) 


I* ****•**«••»•*•»*■»■*»•»*•»•«,•&*■**  end  of  comments  * •<>»***»■& 


BEGIN  MAIN  PROGRAM 

ESTIM8:  PROC  OPTIONS (MAIN) ? 

OCL  ( N » RECLEN  , **REC  * DEL  T'A  » REC ) FIXED  BIN? 

DCL  *)  SAMP  V I XED  BIN  INIT  (0)? 

DCL  SI  GROW  FLOAT  BIN  INI  T(  0.0)1 
DCL  (POINT, DPOINT)  FIXED  BIN? 

DCL  ( I > J , •<  » L » M » L E N_ X ) FIXED  BIN? 

DCL  IY  FIXED  BIN (31)  I N I T i 3 > ? 

DCL  (KGS»;<J.AC»LJf.C«LSOR,KSOB,KLE‘/)  FIXED  BIN? 

DCL  KJA  FIXED  BIN? 

DCL  ( A « B » C , D , Z , MU , E RROR  , T , SNR)  FLOAT  BIN? 

DCL (MEAN, ALPHA)  FLOAT  BIN? 

DCL  < SORER. LMSER)  FLOAT  BIN? 

DCL  JACER  FLOAT  BIN? 

DCL  (rtSOR.BjR)  FLOAT  BIN? 

DCL  ( AMP,FR£Q,PHASE»WT>  FLOAT  BIN? 

DCL  (XLAM.THETA ,XNU»OPTER .LEVER)  FLOAT  BIN? 

ON  ENOPAOE  8=1.0? 

/*  READ  DATA  FROM  CARDS  */ 

GET  OATA  (N,REC_LEM»nr<EC»OELTA  >L JAC » K JAC » LSOR , A SOR t XLEV » SNR , 
MEAN • ALPHA »BSOR»B JR) ? 

PUT  OATA { M » RECLEN , *REC ,DELT  A *L JAC , K JAC , L50R  »KSOR , XLEV , SNR  *i 
mean, aCpha »bsor»sjr) ; 

/*  COMPUTE  ARRAY  DIMENSIONS  »/ 

I=REC_LEN-1 ? 

LEN_X=M  *-DELT  A ? 

J-LEN_X- l J 
K=N-i * 


fflIS  P AGE  IS  BEST  QUALITY  PRACTICABU 
RWM  OUPY  PUftAlSHED  TO  UD C - 
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/»  »/ 

/»  ALLOCATE  STORAGE  ANO  INITIALIZE  ARRAYS  »/ 
/*  */ 


ALLOC: 

begin; 

OCL  S IGNUL (0:1)  FLOAT  BIN? 

DCL  X ( 0 : J ) FLOAT  BIN  I N I T ( ( LEN_X ) 0 . 0 ) * 

DCL  SEOPT ( 0 : I ) FLOAT  BIN? 

DCL  HOP T ( 0 : K ) FLOAT  BIN? 

OCL  M3EO? T ( 0 : I ) FLOAT  BIN  INIT((R£C  LEN)0,0) 
DCL  HLMS  ( 0 J K ) FLOAT  BIN  I N i T ( ( N ) 0 . 0T« 

DCL  SELF'S  ( 0 : I ) FLOAT  BIN! 

ocl  -;gs ; o :k ) float  bin; 
ocl  hsor;o:k)  float  bin; 

DCL  SESOR ( 0 : I ) FLO A T BIN? 

OCL  MSESOR ( 0 : I ) FLO A T n IN 
DCL  MSESOR(OSI)  FLOAT  BIN 
OCL  w J T M p ( 0 : K ) FLOAT  MIN? 

DCL  w J a C ( 0 : K ) FLOAT  BIN  I N 1 1 ( < N ) 0 . 0 ) ♦ 

DCL  HJAC  < 0 : K ) FLOAT  BIN; 

. DCL  S£  JAC  < 0 : I ) FLOAT  BIN; 

DCL  MSE JAC ( 0 : I ) FLOAT  BIN  INIT((REC  LEN)0.0)5 
DCL  HLEV(N)  FLOAT  BIN; 

DCL  WLV(N)  FLOAT  BIN; 

DCL  G t N ) FLOAT  BIN; 

DCL  GOliM(N)  FLOAT  BIN; 

DCL  SELEV (0:1)  FLOAT  BIN? 

DCL  MSELEV ( 0 : X ) FLOAT  blN 
OCL  Y ’>  0 : DEL  T A - 1 ) FLOAT  slN5 
DCL  RXX(OIK)  FLOAT  BIN; 

ocl  ;uxn(o:k)  float  bin; 

DCL  A U T 0 ( 0 : K ) FLOAT  BIN  I N I T ( < *M ) 0.0) 

DCL  W_IN1T(0:K)  FLOAT  BINS 


IN  IT ( (REC  LEv) 0.0) 
INIT ( (BEClLEN) 0.0) 


init ( «rec_len) o.o) ; 


/*  END  OF  STORAGE  ALLOCATION  */ 

/*  INPUT  AMPLITUDE  AND  FREQUENCY  FOR  SKEP  PROBLEM  */ 

GET  LIST ( ASP  »FREO  * I Y ) I 
PUT  SKIP  CaTA (AMP. FREQ*  I Y ) 5 

/*  COMPUTE  SIGNAL  PO'«ER  */ 

SIGPOW=AMP**2/2.05 

/*  INPUT  MU  AND  INITIAL  W VECTOR  */ 


GET  L IST(HU« (W.INIT(I)  DO  1=0  1 
PUT  SKIP  LIST ( HU i (W INIT ( I ) DO 


0 n-1  ) ) ; 

1 = 0 TO  N-1 ) ) ; 


/»  INPUT  OPTIMAL  UNIT  SAMPLE  RESPONSE  */ 

GET  L IS  f ( (HOPT  ( I ) DO  1 = 0 TO  N-  1 ) ) i 
PUT  L 1ST ( (HOPT ( I ) DO  1=0  TO  N-1)); 

END  OF  DATA  ENTRY  AND  IN  I T I AL I Z A T I ON  »•»**«***« 
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SIMULATION  BEGINS  HERE.  ALL  FOUR  ALGORITHMS  AS  SELL  AS  THE 
OPTIMAL  FILTER  FOR  THE  SKEP  PROBLEM  PROCESS  THE  SAME  DATA  AND  ARE  i 
USED  TO  GEN . ESTIMATES  OF  THE  SIGNaL  FOR  SEC  LEN  POINTS.  4HEC 
SUCH  RECORDS  ARE  GENERATED  AMD  THE  SO'JARED  ERRORS  ARE  AVERAGED 
OVER  THIS  ENSEMBLE. 

simulate: 

DO  REC=0  TO  ffREC-1 ? 

/*  initialize  parameters  BEFORE  each  run  */ 
hums =w_i nit: 

WJAC-W_lNl T ; 

hjac=w_init  ; 

UGS  N I T 5 

hsor='*_ini  t ; 

DO  K = Q TO  N-15 

HLEV(K+1)=W  .INIT(K)  J 
«Lv (K> 1 > = W_lNl T (K) ? 

eno; 

x=o.o? 

kgs=o; 

KJA  = 0 ; 

*>  S A M P = 0 ; 

WAX  = 0.0  5 
AUT 0 = 0 . 0 * 

PHASE=6.2831P53*RAN( IY)  » 

RECORD: 

00  1=0  TO 


REC_LEN- 1 } 

/*  GENERATE  SIGNAL  5 DATA 


SAMPLE  */ 


00 


WHILE  ((«SAMP-I)  < LEN  X)5 
SIGNIJL  ( I ) =AMP*COS  { t-'REO*^S  am®  + PH  A SE  ) * 

POINT  = M00  ( (4SAMP+LEN  X)  »LEN  X); 

X (POINT ) =S  IGNUL  ( I ) ♦* NOISE  ( ACTPHA  * DELT  A * #S AMP  * Y » I Y ) * 
SURT (SIGPOW/SNR) >MEAN • 

AUTOCORR : 

DO  J = 0 TO  N-l  WHILE  ( *SAMP>=J)  ? 

M = MOO  ( PO  I N'T  - J *LEN_X  • LEN  X)! 

RXX ( J) =RXX ( J)  <M  4 S A m p - j ) 7 ( 4S AMP-J ♦ 1 > -X (POINT) * 

X (M)  / USAMP-J*!  ) ; 

IF  (4 SAMP-DELTA) >= J 
THEN  DO 4 

OPO  INT  = M00  ( PO  I rl  f-DELT  A- J+LEN_X  ♦ LFN..X  ) ? 

M = = SAMP-DELTA-J  5 

AUTO ( J) =AU  TO ( J)  <W  (Mf l ) ‘X (DPOINT) *X (POINT) / 
( M ► 1 ) * 

end; 

END  AUTOCORR? 


4 S A M P = 4 S A M P + l 5 


END? 


TlilS  PAGJP  rc 

f^°M  ou ry  tnJ,^ST  Quality  rr. 

^^SHJSD  xo  w **<>*10*^ 
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/*»«•»«  »•**!»»•»  •*•*•**»  *•*!»**#  a*/ 

/*  */ 
/*  UPDATE  UNIT  SAMPLE  RESPONSE  FOR  JAC0«I  */ 
/*  RELAXATION*  SOP  ALGORITHM  AND  LEVINSON  */ 


/*  ALGORITHM, 
/* 


*/ 

*/ 


/■»  30R  ALGORITHM  WITH  VARIABLE  ITERATION  */ 

/*  STEPS.  */ 

IF  MOO  1 1 .KSOR) =0 
THEM  DO* 

El  S 0 P - \t  6 $ ? 

SO h : 

no  K = 0 TO  LSOR-1* 

IF  KGS>N~1 
THEN  K G S = 0 ; 

w G S ( x G S ) = ( 1 • - S S 0 K ) * w G S ( !<  G 5 > * P X X ( 0 ) / B 3 0 R ? 

DO  J=0  TO  N-li 
M-ABS ( K G S ~ J ) 5 
IF  J *• = K G S 

THEN  wGS  (KGS)  =WGS  ( KGS ) -RXX { M ) *WGS ( J ) ? 

END  * 

wGS  (KGS)  =<W3S  (KGS)  ♦AUTO  (KGS)  ) "'BSGR/RXX  ( 0 ) 5 
KGS=KGS+1 5 

end  sor; 

end; 

/*  JACOBI  RELAXATIN  METHOD  WITH  VARIABLE  * / 

/»  ITERATION  STEPS,  >/ 

IF  MOD  ( I .K.JAC)  -0 
THEM  DO* 

H^  Ac  — W U A C f 
W-JTMPSWJAC* 

JACOBI : 

DO  K=0  TO  LJAC-1J 
IF  KJA>N-1 
THEN  K J A = 0 5 
WJAC(KJA)=0.e» 

DO  J-0  TO  N-li 
M=ABS (KJA-J) J ' 

WUAC  iKJA)  = W J A C ( K J A ) -RXK  (M)  »WJ  IMP  ( J ) 5 
END » 

WJ«C  (KJA)  -=WJTmP  (KJA)  ♦ BJB*  ('J.JAC  (KJA)  ♦AUTO(KJA)  ) 
/ RXX  (0)5 

KJA-KJ A*  I ; 


END  JACOB  I * 


END? 
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/*  LcVINSONS  ALGORITHM  '>/ 

IF  MOO(I«KLcV)sO 
T hffc'N  DO? 
hlev=wlv; 

/*  INITIALIZE  VECTORS  */ 

RXXN=RXX/RXX (0) } 

WLVsO.OJ 
G = 0 • 0 * 

WLV ( 1 ) = AU TO ( 0) /RXX  (0)  5 
G ( 1 ) =-RXXN (1)5 
XL AM= 1 ,-kXXN i 1 ) 

LEVINSON: 


DO  K= 1 TO  N-i; 

TKETA  = AUT0(O/RXX('J)  ? 

IF  (KMX=(N-1) 

THEN  XNU  = -RXXN  (KM  ) ♦ 

DO  J=1  TO  k; 

ThETa=Th£TA-WLV ( J) *RXXN(K-JM) 5 
xnu=xnu-rxxn ( jj  »o  ( j) ; 
end; 

DO  J=1  TO  k; 

WLV { J) =WLV ( J) ♦ T HETa/XLAM*G  ( J ) 5 
GDUM  < J*  I ) =G  ( J ) *XNU/XLAM*S  ( K-*  J*  1 ) 5 

end; 

G( 1 ) =xnu/xlam: 

DO  J = 2 TO  :<M» 

G( J) = GDUM { J) } 

end; 

WLV  (KM  ) sTHETA/XLAMI 
xlam-=xlam-xnu«>2/xlam; 

end  levinson; 

end; 


/*  END  OF  WEIGHT  VECTOR  UPDATE 


/*  */ 
/*  COMPUTE  SIGNAL  ESTIMATES  */ 


/*  */ 


*/ 


SEOPT i I)=0.0( 
SELMSt I)=U.O; 
SESOR ( I ) =0.0  5 
SEJAC ( I ) = 0,0  » 
5ELEV ( I ) =0.0  ; 


SIGFST : 

00  J=0  TO  NMt 

OPOlNT=MOO  (POINT-OELTA-Jt-LEN  X -LEN  X)J 
SEOPT ( I ) =SEOPT ( I ) ♦PORT ( J) (OPOINTTJ 
SELMS  ( I ) =SELMS  ( I ) ♦ nLMS  ( J ) »X  (DPOINT)  ; 
SEJAC  ( [ ) =SE  J AC  ( I ) +HJAC  ( J ) *X  (DPOINT)  *, 
SESOR ( I ) =SESOR ( I ) + HSOR ( J) *X (OPOINT)  ; 
SELEV  ( I ) =SELE V ( 1 ) ♦hLEV  ( J*  i ) ■>  x ( OPOI NT  ) ; 

END  sigest; 


/*  END  OF  SIGNAL  ESTIMATION  */ 
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/*  »/ 

/*  UPDATE  I.MS  WEIGHT  VECTOR  */ 
/'*  -V 


£RROR=X  (POINT)  -SELMS  ( I ) i 
UPDATE  : 

00  J = 0 TO  N-i; 

DPOINT=MOO(POlNT"Dfc.LTA-J*LEN  X.LEN  X); 
HLMS< J) = HLM5 ( J) ♦2«*MU*ERR0R*X (QPOTnT) 5 
END  UPDATE! 


'/*  */ 

/*  COMPUTE  MSE  FOR  ALL  ALGORITHMS  */ 
/*  */ 

/ 4»  -»  ■»  1>  » '><■»*&»»  » »<•*■<»•» 

UPTER= ( 5IGNUL ( I ) -StOPT ( I > ) »*2; 
LMSER=  ( S IG.mUL  ( I ) -SELMS  ( I ) ) **2  f 
SORER* (SICNUL ( I ) -SbSOR ( I } ) **?5 
JACER=(SIGNUL(I)-SEJAC( I) )**2; 
LEVER= (SIGNUL(I)-SELEV (I) )»*25 
MSEOPT ( I ) =MS£OPT ( I ) ♦OP  TER/ APEC  * 
MSF.LPS  { I ) =MSELMS  ( I ! + I_MSE«/*REC  ! 
PStJAC ( I ) =mSEJAC ( I ) ♦JACER/aREC* 
MSESOR ( I ) sMSESOR ( I ) ♦ sorer/*rf.c; 

MSELEV  ( I ) sMSELEV  ( I ) +LEVF.R/ s*EC  i 


END  RECORD I 
END  simulate; 


END  OF  SIMULATION  LOOP  »»■»*!>•*  »*#*#*»***** 

£ ■»•  •&•!»*  tt »«■»•»**•»•»  *«••»■» ««■»« «<»»»'»*  -55  *«*■>;  «•*•»«  *■»«• 

THTS  SECTION  TAKES  THE  RESULTS  OF  THE  PROCESSING  0^  THE  LAST 
DATA  RECORD  AND  PREPARES  AND  PLOTS  THE  PROGRAM  OUTPUT 


out  ; 

do  i-o  to  reclen-i; 

PUT  SKIP  EdIT(SIGNULU)  •SEOPT(I)  ,SELmS(I)  ,SCJAC(n  » 
SESOR ( I ) * SfcLE v < I ) » MSEOpT ( I ) *HSf LMS ( I ) >MS£JAC ( I ) » 
MSESOR ( I ) VmScLEV (I))((llJE(13«E))5 

END  out; 

»»•»**#**«»»»« End  MAIN  PROGRAM  • asa-xn**'***#****--****’ 


i 


aaaaaaaaaaaaaaaaaa-a’aaaaatfaaaaaaaaaaa-wa-aaaaaaaa-.jaa-aaaaaaaaaatfa-a**-;: 

XNOISE  GENERATES  AN  AUTOREGRESSIVE  SEQUENCE  WHEN  CALLED 
RECURSIVELY.  XNO ISE=SUM (( ALPHA **K )«/( X ) ) K = 0 . DEl.'IA-i,  WHERE 
Y ( K ) ARE  INDEPENDENT,  IDENTICALLY  !.)  I STn  I bU  I TED  RANDOM  VARIABLE*: 
TO  SAVE  EXECUTION  TIME*  NOISE  SAMPLE  IS  ACTUARY  COMPUTED  IN  A 
RECURSIVE  FASHION. 

XNOISE : PROCIALPHA, DELTA, #SAMP,Y,IY)  RETURNS (FLOAT  BIN); 

dcl(alpha,xold,yolo>  float  bin; 

DCL  XNEW  FLOAT  BIN  STATIC, 

OCL <OELTA,*SAMP,K, POINT)  FIXED  BIN: 

OCL  T.Y  FIXED  9 IN  (3D  i 

dcl  v'*>  float  bin; 

if  ft s AMR  = a 

THEN  Q0»  /*  INITIALIZE  SEQUENCE  ON  FIRST  CALL 

xnew=o.o; 

00  K=0  TO  DELTA-l ; 

Y (K) =GAUSS ( IY)  ; 

xn£w=alpha*xnew>y  ( k ) ; 
eno; 
end; 

else  oo; 

xold=xnew; 

POINT=MOO(ftSAHP-OELTA-l  , DELTA)  ; 

YOLD=Y (POINT) ; 

Y (POINT) -GAUSS ( I Y) ; 

XNEW=ALPHA*X0LD+Y ( POINT) -YOLO* ALPHA **DELT A ; 

end; 

/*  SCALE  SO  THAT  VAP(X0LD)=1  »/ 

XOLO=XNFw*  ( I -ALPHA**  (2*  ( DELTA- 1 ) ) > / ( 1-ALPHA**2)  ; 

RETURN (XOLO) ; 

ENO  xnoise; 

GAUSS  GENERATES  NORMAL (0,1)  RANDOM  VARIABLES  RY  TAKING  THE  SUM 
OF  12  INDEPENDENT  UNIFORM (0,1)  RANDOM  VARIABLES  GENERATED  BY  TM 
POWER  RESIDUE  mFTHOO.  SINCE  THE  MEAN  OF  J(0.1)=l/2  AND  THE  VAR 
IANCE  OF  U(0,l)=l/i2«  THE  SUM  OF  12  OF  THESE  INDEPENDENT  RANDOM 
VARIABLES  HAS:  m£ AM ( SUM ) =SUM ( MEANS ) =b  AND  VAR(SUM) =SUM(VAR)=l. 
THEREFORE,  THE  R.V.  X = SUM-6  HAS  M E A N = 0 AND  VAPTANCF=1  AND  IS 
APPROXIMATELY  NORMAL  BY  THE  CENTRAL  LIMIT  THEOREM. 

44444#*fl*«444444444*44'444444444ittt«>4  44  44  5*44»44>4->444*4J9  4»4444  4 

( NOS  I ZE ,NOF I XEOQVERFLOW ) : 

GAUSS:  PROC(IY)  RE  TURNS (FLOAT ) ; 

OCL  I FIXED  BIN: 

DCL  IY  FIXED  BIN  (31) ; 

DCL  A FLOAT  BIN? 

a=o.o; 

DO  1=1  TO  12; 

IY=IY*6SS395 

^ ^TH£N<  IY  = IY,?1A7A836A7>1;  IS  BEST  QUALITY  ?RACTIC4flU| 

A=IY»0,4656613E-9+A;  2B0ME2  fURkisw  Xu  uDG 

end;  — 

RETURN(A-A.O) ; 

END  GAUSS; 
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SUBROUTINE  HAM  GENERATES  UNIFORM  (0,1) 
USING  THE  POwER  RESIOUE  METHOD. 


HANOOM  VARIABLES 


( NOS  I ZF  » NOF I XEOOVERFLOW ) : 

RAN*.  PROC(IY)  RETURNS  (FLOAT  BIN)  5 
OCL  IY  FIXED  BIN  (31.)  i 
OCL  A FLOAT  8 IN  ? 
rY=rr^65Sjy; 

IF  IY  < 0 

THEN  IY=IY*21  ^74<336*v7  + I 
A=IY»y.*&566l3E-9» 

RETURN ( A) ; 

ENO  RAM  5 


END  ALLOC  I 
END  F.STIM8? 


1 
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D.2  Computation  of  Eigenvalues  and  Eigenvectors 

The  theoretical  bounds  derived  in  Chapters  II  and  III 
require  knowledge  of  the  autocorrelation  matrix  and  its  eigen- 
values. Expressions  for  the  autocorrelation  matrix  and  vector 
for  the  SKEP  problem  were  derived  in  Chapter  V.  The  computer 
program  described  here  determines  and  its  eigenvalues  and 

eigenvectors  as  well  as  the  regular  and  uncoupled  optimal  unit 
sample  response  vectors.  The  uncoupled  initial  unit  sample 
response  is  also  computed.  The  program  was  run  on  an  IBM  370/165 
computer  and  is  written  in  the  PL-I  language. 

D.2.1  Specifications  of  the  Problem. 

• 

The  specifications  of  the  problem  are  discussed  in  Section 
6.1  and  are  summarized  below: 

1.  Single  frequency  sinusoidal  signal  with  random  phase 
uniformly  distributed  on  [— tt,tt]  specified  by  equation 
(5.1). 

2.  Zero-mean  Gaussian  autoregressive  noise  sequence  with 
autocorrelation  function  given  by  equation  (5.18). 

3.  Filter  length  N * 16. 

4.  Parameter  values  for  data  generation  summarized  in 
Figure  5.1. 


2: 


D.2.2  Program  Structure 


The  structure  of  the  program  is  illustrated  in  Figure 


D.2.  Each  program  segment  is 

described  below: 

1.  EIGEN  — 

Main  program. 

Contains  program  segments  ALLOC 

AUTO-CORR,  USR,  MMSE  and  OUTPUT.  Calls  pro- 

cedure  MSDU. 

INPUTS : 

N — 

Filter  length. 

DELTA  — 

Delay  value  n^. 

ALPHA  — 

Autoregression  parameter  for 

noise  sequence. 

SNR  — 

• 

Signal-to-noise  ratio. 

AMP  — 

Amplitude  of  signal. 

FREQ  — 

Frequency  of  signal  (radians) . 

H-INIT  — 

Initial  unit  sample  response 

vector. 

2 . ALLOC  -- 

Allocates  required  storage. 

INPUTS : 

J 

N — 

Filter  length. 

3 . AUTO-CORR 

Computes 

autocorrelation  matrix  and  vector 

INPUTS : 

N — 

Filter  length. 

DELTA  — 

Delay  value  n. . 

Autoregression  parameter  for 
noise  sequence. 


ALPHA 


2: 


Figure  D.2  Structure  of  Program  EIGEN  — for  the 
Computation  of  the  Eigenvalues  of 
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SNR  — 

AMP  — 

FREQ  — 

OUTPUTS : 

MAUTO  — 

RXX  — 

VAUTO  — 

4 . MSDU  — Computes  eigenvalues  and  eigenvectors.  Available 
in  the  IBM  System  360-370  Scientific  Subroutine 
Package . 

INPUTS: 

N — . Filter  length. 

MAUTO  — Autocorrelation  matrix. 


Signal-to-noise  ratio. 

Amplitude  of  signal. 

Frequency  of  signal  (radians). 

Autocorrelation  matrix. 

Vector  which  determines  Topelitz 
autocorrelation  matrix. 
Autocorrelation  vector. 


OUTPUTS : 

MAUTO  — Matrix  contains  eigenvalues  along 

the  diagonal. 

MODAL  — Modal  matrix  of  the  system. 

5.  USR  — Computes  optimal  unit  sample  response,  uncoupled 

optimal  unit  sample  response  and  uncoupled  initial 
unit  sample  response.  Contains  segment  UCVAR. 
Calls  procedure  MMULT. 

INPUTS: 

MAUTO  — Matrix  containing  eigenvalues 

along  diagonal. 

MODAL  — Modal  matrix  of  the  system. 


J 


I 
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N — Filter  length. 

HINIT  — Initial  unit  sample  response. 

OUTPUTS : 

H — Optimal  unit  sample  response 

vector . 

HUC  — Uncoupled  optimal  unit  sample 

response  vector. 

HIUC  — Uncoupled  initial  unit  sample 

response  vector. 

6.  MMSE  — Computes  MSE  produced  by  the  optimal  filter. 

INPUTS: 

MAUTO  — Matrix  containing  eigenvalues 

along  the  diagonal. 

ViUC  — Uncoupled  optimal  unit  sample 

response  vector. 

AMP  — Signal  amplitude. 

OUTPUTS : 

MSE  — MSE  produced  by  the  optimal 

filter. 

7.  OUTPUT  — Outputs  results  of  program. 

INPUTS : 

MAUTO  — Matrix  containing  eigenvalues 

along  the  diagonal. 

H — Optimal  unit  sample  response 

vector. 

HUC  — Uncoupled  optimal  unit  sample 

response  vector. 


28] 


I 

HINIT  — 

HIUC  — 

RXX  — 

N — 

DELTA  — 
ALPHA  — 

AMP  — 

FREQ  — 

SNR  — 

MSE  — 

OUTPUTS : 

N — 

DELTA  — 

AMP  — 

FREQ  — 

ALPHA  — 

MSE  — 

MAUTO  (1,1)  — 

H — 


Initial  unit  sample  response 
vector. 

Uncoupled  unit  sample  response 
vector. 

Vector  which  determines  the 
Topelitz  autocorrelation  matrix. 
Filter  length. 

Delay  value  n^ . 

Autoregression  parameter  for 
noise  sequence. 

Signal  amplitude. 

Signal  frequency  (radians) . 
Signal-to-ijoise  ratio. 
Mean-square  error  produced  by 
optimal  filter. 

Filter  length. 

Delay  value  n^. 

Signal  amplitude . 

Signal  frequency  (radians) . 
Autoregression  parameter  for 
noise  sequence. 

MSE  produced  by  optimal  filter. 
Eigenvalues  of  autocorrelation 
matrix. 

Optimal  unit  sample  response  - 
Initial  unit  sample  response. 


HUC 


MMULT  — 
INPUTS : 


OUTPUTS 

UCVAR  — 

INPUTS : 


OUTPUTS 


Uncoupled  optimal  unit  sample 
response  - Uncoupled  unit  sample 
response. 

RXX  — Vector  which  determines  the 

Topelitz  autocorrelation  matrix. 
Multiplies  an  NxN  matrix  and  an  N-vector. 


MATRIX  — 
VI  — 

N — 

K — 


Matrix  for  multiplication. 
Vector  for  multiplication. 
Filter  length. 

K-0  gives  MATRIX’ VI. 

K=1  gives  MATRIX7 • VI. 


VR  — Vector  result  of  the  multiplica- 

tion. 

Solves  system  of  equations  in  terms  of  uncoupled 
variables . 


MAUTO  — 


VAUTO  — 
N — 


Matrix  containing  eigenvalues 
along  the  diagonal. 
Autocorrelation  vector. 

Length  of  filter. 


HUC  — 


Uncoupled  optimal  unit  sample 
response  vector. 
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/*  *>/ 

/*  ThIS  PROGRAM  COMPUTES  THE  EIGENVALUES  OF  T HF */ 
/»  AUTOCORRELATION  MATRIX  FOR  ThE  SKEr  PROBLEM  */ 
/*  The  normal  and  UNCOUPLED  OPTIMAL  Unit  sample*/ 
/»  RESPONSES  AND  THE  UNCOUPLED  INITIAL  UNIT  '*/ 

/*  SAMPLE  RESPONSE  ARE  ALSO  COMPUTED  AS  WELL  */ 

/*  AS  THE  AUTOCORRELATION  VECTOR.  */ 

/»  */ 

/ v*  # sr  *>  ■5'  4*  £*  O ■»>  ->  ft  -Jr  & <!r  <•  Sr  & it  # ■*>  <e  v>  >5*  it  ■{>  ->  .;{  <}»  •>  / 

EIGEN  : PROC  OPTIONS (MAIN) ; 

OCL  ( AMP  » FREQ. SNR .ALPHA  * T ) FLOAT  8 IN  5 
OCL  (DELTA* I • J.K ,M,N)  FIXED  BINS 
OCL  (RS3.RNN.MSE)  FLOAT  BIN. 

/*  READ  DATA  FROM  CAROS  */ 

GET  DATA ( N * OELT  A » AMP  * FREQ  j T » ALPHA , SNR)  S 

/*  ALLOCATE  STORAGE  */ 

ALLOC : 

begin; 

DCL  MAUTO(N.N)  FLOAT  2IN? 

OCL  VAUTO(N)  FLOAT  JIN? 

OCL  H ( N ) FLOAT  BIN  I N I T < ( N ) 0 . 0 ) ? 

OCL  HUC(N)  FLOAT  BIN  TnI  T ( (N)  0 . 0).J 
OCL  MODAL (N.N)  F i OAT  BIN  7 
DCL  HINIT(N)  FLOAT  SIN  I N 1 7 ((•'•')  0 « 0 ) J 
DCL  HIUC(N)  FLOAT  SIN  I N I f ( iw) 0 , 0 ) * 
OCL  RXX(N)  FLOAT  BIN. 

DCL  MSOU  ENTRY  ((*♦*)  F L 0 A T BIN;  (*•;*) 
FIXED  3IN)5 


'LOAT  BIN, FIXED  ?IN, 


/*  END  OF  STORAGE  ALLOCATION  */ 

/*  INPUT  INITIAL  UNIT  SAMPLE  RESPONSE  */ 

GET  LIST ( (HI  MIT  (I)  DO  1 = 1 TO  N) ) { 

/*  COMPUTE  AUTOCORRELATION  MATRIX  AND  VECTOR  */ 

AUTO.CORR: 

DO  1=1  TO  NS 
DO  J=1  TO  I ; 

K = A 8 S ( I -J)  ; 

RSS  = 0 . S* i AMP«»2) »COS ( b . 233 1 B5*FRE<)*7  *K ) ; 

IF  DELTA 

THEM  R NN  = (ALPHA**!'-)  * ( ( 1 . -ALPHA  ( ?«OELTA-c*K  ) )/ 

( I ,-ALPHA*»2) ) * (AMP>b?>/(2,*SNR)  5 
ELSE  RMN  = 0 . 0 S 

mauto ( i . j) =rss*»mn; 

MAUTO  ( J.  I ) = M A U-T 0 ( I 7 J)  } 
end; 

VAUTO ( I ) =0.5* ( AMP* *2) *C0S ( 6 . 23  3 1 85*FREO*T* (OELTA+I-1 ) ) T 
r x x ( i ) =mauto( i 7 1)  ; 

ENO  AUTO,CORR5 

/*  COMPUTE  EIGENVALUES  AND  EIGENVECTORS  */ 
k = o ; 

CALL  MSDU (MAUTO* MODAL ;M»K> ; 
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/*  DETERMINE  uncoupled  initial  unit  sample  response  */ 
k=i  ; 

CALL  MMULT (MODAL* MINI  T » H I UC  * N * K ) » 

/*  SOLVE  EONS  IN  TERMS  OF  UNCOUPLED  VAR  I ABLES  */ 

UCVAH*. 

00  1=1  TO  n; 

00  J=1  TO  N * 

MUCH  ) =HUC  ( I ) +MODAL  ( J * I ) *VAUTO( J)  ; 
end; 

HUC ( I ) = MUC ( I ) /MAU TO < I » I ) * 

END  UCVARJ 

/*  find  SOLUTION  IN  TERMS  OF  COUPLED  VARIABLES  */ 

K = 0 1 

CALL  MMULT ( MODAL  *HUC  * H * N *K ) * 

/*  FIND  MINIMUM  MSE  */ 

MSE=(AMP**2) *0.5? 

MMSE  ? 

00  1=1  TO  N? 

MSE=MSE-MAUTO ( I * I ) *MUC ( I ) **2  ? 

END  MSSE? 

/*  OUTPUT  RESULTS  */ 

HsHINlT-HI 
HUC=H I UC-MUC • 

PUT  SKIP  EDIT (N*DELTA»ALPhA *AmP)  { ( 2 ) F C S I . (2)E(13*6) ) ? 
PUT  SKIP  EDIT<FRE0.T*5NHt.MSE>  ((*>£{  13*6)  > ? 

DO  1=1  TO  N? 

PUT  SKIP  EDIT  'MAUrO(I»I>  *H(I)  *HIJC<!>  ,RXX(I>  ) ((A)E(13* 
ENC? 


6)  Vi 


/*  MMULT  MULTIPLIES  AN  N X N MATRIX  BY  A VECTOR*/ 


MMULT  : PROCt  MATRIX*  VI  «V.R,M,K)  i 

DCL  MATRIX!*.*)  FLOAT  BIN* 

DCL  Vl(*)  float  bin; 

DCL  V R ( * ) FLOAT  BIN  5 
DCL  (I.J.N.K)  FIXED  BIN? 

MULT:  : 

DO  1=1  TO  N? 

VR ( I ) =0 • 0 * 

DO  J-l  ro  Nl 
IF  K = 0 

THEN  VR(I)sVR(I)*MATRIx(I«J)*VI  (J)  i 
ELSE  VR(I)=VH(I)+MATRIX(J*I)*V1(J) i 

ENOi 

END  mult; 
return; 

END  MMULT; 

end  alloc; 

END  EIGEN; 


* »•»«•*****  ->««»»«««*«»<*.«/ 
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D. 3 Program  for  Evaluation  of  Theoretical  Bounds 

The  program  described  in  this  section  was  used  for  the 
evaluation  of  the  theoretical  bounds  derived  in  Chapters  II 
and  III.  In  addition,  the  plots  of  the  theoretical  bounds  and 
of  the  results  of  the  results  of  the  simulation  were  made  using 
this  program.  A PDP  11/45  computer  with  storage  tube  graphics 
display  was  used  and  the  data  generated  by  the  simulation  pro- 
gram and  the  eigenvalue  program  were  stored  as  data  files  on 
the  PDP  11/45  magnetic  disk  for  access  by  this  program.  The 
program  was  written  in  the  FORTRAN  IV  language  and  operates  in 
an  iteractiva  fashion. 

D.3.1  Program  Structure 

The  structure  of  the  program  is  illustrated  in  Figure 
D. 3.  Each  program  segment  is  described  below: 

1.  THEO  — Main  program  for  evaluation  of  theoretical  bounds 
and  display  of  data.  Calls  Subroutines  LMS , 
JACOBI,  SOR,  DDATA  and  PLOTIT. 

INPUTS : 

N — Filter  length 

DELTA  — Delay  value  n^. 

ALPHA  — Autoregression  parameter  for 

noise  sequence. 

AMP  — Amplitude  of  signal. 

Frequency  of  signal  (radians) . 


FREQ 


AMSE 


'EKPLT 


CRCNTL 


Structure  of  Program  THEO  — For  the  Computation  of 
Theoretical  Bounds  and  for  Display  of  Data. 


SNR 


EMIN  — 
EIG  — 


H — 


HUC 


RXX  — 


OUTPUTS : 


Signal-to-noise  ratio. 

MSR  produced  by  the  otpimal  filter 
Eigenvalues  of  the  autocorrela- 
tion matrix. 

Optimal  unit  sample  response 
vector  — Initial  unit  sample 
response  vector. 

Uncoupled  optimal  unit  sample 
response  - Uncoupled  initial 
unit  sample  response. 

Vector  which  determines  the  Tope- 
litz  autocorrelation  matrix. 


Same  as  inputs . 

2.  LMS  — Computes  theoretical  bound  on  MSE(n)  for  the  LMS 
Gradient  Algorithm. 

INPUTS: 


U — 

N — 

NP  — 
HUC  — 


EIG  — 


EMIN  — 
I — 


y value  for  LMS  Gradient  Method. 
Filter  length. 

Number  of  samples  to  be  computed. 
Uncoupled  optimal  unit  sample 
response  - Uncoupled  initial  unit 
sample  response. 

Eigenvalues  of  autocorrelation 
matrix. 

MSE  produced  by  the  optimal  filter 
Row  of  array  X in  which  results 





' 1*  !. 


r 

288  i 


are  to  be  stored. 

| 

OUTPUTS : 

X ( I , * ) — Row  of  array  containing  samples  of 

the  theoretical  bound  for  the  LMS 
Gradient  Algorithm. 

3.  JACOBI  — Computes  theoretical  bound  on  MSE(n)  for  the 

Jacobi  Relaxation  Method. 


INPUTS : 

B — 


N — 

NP  — 
HUC  — 


EIG  — 


EMIN  — 


I — 


NS  — 


OUTPUTS : 

X(I,‘)  — 


Parameter  6 for  the  Jacobi  Re- 
laxation Method. 

Filter  length. 

Number  of  samples  to  be  computed. 
Uncoupled  optimal  unit  sample 
response  - Uncoupled  initial  unit 
sample  response. 

Eigenvalues  of  the  autocorrelation 
matrix. 

MSE  produced  by  the  optimal  fil- 
ter. 

Row  of  the  array  X in  which  re- 
sults are  to  be  stored, 
k value  for  the  Variable  Itera- 
tion Step  Method. 

Row  of  array  containing  samples 
of  the  theoretical  bound  for  the 


I 


Jacobi  Relaxation  Method 


4.  SOR  — 


Computes  theoretical  bound  on  MSE(n)  for  the 
SOR  Algorithm.  Calls  function  AMSE. 
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INPUTS : 


B — 
N — 
NP  — 
H — 


RXX  — 


EMIN  — 
I -- 


NW  — 


NS  — 


OUTPUTS : 


Parameter  3 for  the  SOR  Algorithm,  j 
Filter  length. 

Number  of  samples  to  be  computed. 
Optimal  unit  sample  response  - 
Initial  unit  sample  response. 

Vector  which  determines  the 
Topelitz  autocorrelation  matrix. 

MSE  produced  by  the  optimal  filter. 
Row  of  the  array  X in  which  re- 
sults are  to  be  stored. 

1 value  for  the  Variable  Iteration 
Step  Method. 

k value  for  the  Variable  Iteration 
Step  Method. 


X(I,*)  — Row  of  array  containing  samples 

of  the  theoretical  bound  for  the 
SOR  Algorithm. 

5.  DDATA  — Reads  data  file  produced  by  the  Simulation 

program. 


INPUTS : 

Filename  for  the  data  file  containing  the  simula- 
tion results. 

NC  — Particular  variable  which  is  de- 


sired. 
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Row  of  the  array  X in  which  re- 
sults are  to  be  stored. 


OUTPUTS : 

X (I , • ) — Row  of  array  containing  the  de- 

sired data  from  the  results  of 
the  simulation. 

6.  PLOTIT  — Plots  data  on  storage  tube  graphics  terminal. 

Calls  subroutines  TEKPLT  and  CRCNTL. 


INPUTS : 


NP  — 

NV  — 

NS  — 

PVAR  — 

X(I,J)  — 

NS TART  — 

NFIN  — 

IDEV  — 


number  of  points  to  be  plotted 
per  curve. 

Number  of  curves  to  be  plotted  on 
same  graph. 

NS=0  for  manual  scaling,  NS=1 
for  automatic  scaling. 

Array  containing  indices  of  the 
rows  of  array  X to  be  plotted. 
Array  containing  data  to  be 
plotted. 

Number  of  point  at  which  plot  is 
to  begin. 

Number  of  point  at  which  plot  is 
to  end. 

Unit  number  of  graphics  device. 
Vertical  scale  factor  if  manual 
scaling  used. 


YS 


OUTPUTS 

AMSE  — 

INPUTS : 


OUTPUTS 

TEKPLT  — 
INPUTS : 


OUTPUTS 


YINT  — Y-axis  intercept  if  manual  scaling 

used. 

Plot  of  the  contents  of  the  selected  row(s)  of 
the  array  X. 

Computes  MSE  for  the  SOR  Method  given  the  value  of 
the  unit  sample  response  vector. 


HW  — 
RXX  — 


EMIN  — 
N — 


Unit  sample  response  vector. 
Vector  which  determines  Topelitz 
autocorrelation  matrix. 

MSE  produced  by  optimal  filter. 
Filter  length. 


AMSE  — MSE  using  the  unit  sample  response 

vector  HW. 

System  subroutine  for  vector  graphics. 


M — 

M=1  for  visible 

vector,  M=0  for 

invisible  vector 

• 

1X1  — 

X coordinate  of 

first  endpoint. 

IY1  — 

Y coordinate  of 

first  endpoint. 

1X2  — 

X coordinate  of 

second  endpoint. 

IY2  — 

Y coordinate  of 

second  endpoint. 

IDEV  — 

Unit  number  of  graphics  device. 

Vector  drawn  from  (1X1,  IY1)  to  (1X2,  IY2) . 


I ‘ 


I - 
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9.  CRCNTL  — System  subroutine  for  generating  carriage  control 
characters  for  graphics  device. 

INPUTS : 

IDEV  — Unit  number  of  graphics  device. 

OUTPUTS : 


ISP  — 

Character 

for 

single 

space 

IDSP  — 

Character 

for 

double 

space 

IPLUS  — 

Character 

for 

+ • 

I DOLL  — 

Character 

for 

$. 

I 


non  nnn  n nn  nn  n n no  non  non 


Lais  PAQB  IS  BEST  i^Uimi B'""* iJ 

g CCi  Y K JKalSHED  TO  DD 
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D.3.2  Source  Program 

INTEGER  DELTA 
COMNCN  *(8,512) 

dim  ensign  NVAR(fi)#NCRn(e),Kjc(ie)»Ki<i)fEiG(ias,nx)((i6) 
OAT 4 8LANK/4H  / 

ASK  FOR  KEYBOARD  CEVICE  N’UNCFR 

WRI7E(S,2600) 

REACH,  1000)  ICEV 


ASSIGN  DISK  F!U  CCNTAIKING  F l C F N V A l U !:  S TO  i.  NTT  t 

B WRITE (5/2039) 

PAUSE 

DEFINE  FILE  1 ( 1 8 , 2 / L , INI'X) 

READ  EIGENVALUE  DATA  FRCM  DISK 

REACH  U)  RN'/RCEL,  ALPHA,  APP 
R F A C H *7  ) FREE, 7, 3NF, ENIN 
,»js  1ST  UN) 

CFLTAaINrfROEL) 

HO  10  t » l • N 
!<-!♦? 

RFAPH'K)  EIG<I>,KI)/HUC(I)/FXX<IT 
je  CONTINUE 
ENO  FILE  1 

WRirr  eigenvalue  cata  tc  linf  printer 

WRITS (5,2 100)  N / CELT  A,  AL  PH  A , A HP  , FFT:  G , T , SNR  , EVIN 
‘•RITE  (S,P?S!0) 

DO  IS  Isi/N 

HSITG(5,23Be)  E IG  ( I ) , H ( I)  , i-UC  ( I ) , RXX  ( I ) 

15  CONTINUE 
END  FILE  5 

ASK  WHIO  CURVES  Tf.  SS  GENERATED 

17  WRITE <3/2400) 

REACH, 1002)  (NCRC(  I)/ 1st #9) 

ASK  n POINTS  TC  8c  GENERATEC 

WRITE (6,2*00) 

REAC(6,  I 102)  >NF 

CALL  APPROPRIATE  SUBROUTINES  1C  GENERATE  CURVES 


DO  60  Tat, a 

IF (NCRCCI) ,£G,e)  GO  TC  60 
KaNORC(T) 

UC  TO  (2?,  J?,  4c?,  OF,  55),  K 
C A U.  L y S ( I , N , h P , ! I C , F 1 G , F N 1 N ) 
GC  TO ' 69 


72 


n r»  n 


3?  CALL  JA(*C3Id/N.NP,H  C.EIG,R*X,EPIN) 

GC  T 0 50 

4i?  CALL  SCR(I.N/NF/h/RXX,F»>lN) 

GC  TO  60 

3 ‘Z  CALL  ODATAJI/NF) 

GC  TO  60 

55  OC  37  INa  l / K F 
X(I»TN)«BMN 

57  CONTINUE 

60  CONTINUE 

PLOT  DATA 

30v1  aRHE  d,  2*32) 

RT  AC  ( 6 , ! vltfG  ) ( N V A R ( I)  / I s [ , A ) 

■<RI  TP  (6/?7fd) 

RE A 0(6/1  IPO)  N 2 7 AR T • NFI N 
URtTE (S,2?50) 

REA0(6, 107|Z‘)  NS 

CALL  PLO TIT (5 12, 8, NS, NVAP, N START / NF IN, ICEV) 

URITE (6,2800) 

READ (6, 1300)  iN C F T 
IF(\CPT.ECi.  1 ) GC  TO  300 
WRIT1"  (6,^652) 

KFAP  t.«#  ti?Off  ) VCF  r 
IF  (NCP  r >:'?■.  j ) GC  TO  17 
URTTF.  f6/2S03) 

«F  AC  (5,  • €19»G  > NCFT 
IFCNC»’?.E«.1)  GC  TO  5 
toot;  F OR!'  A T d a » 6 » 1 1 » 2 X ) ) 
t 1 OS  rCRV  A T C i x , 13 , 2X , 13 ) 

20  OK  FOR*' A T < I H , 'ASSIGN  EIGENVALUE  C 1SN  FILE,  EX  APPL  P i ' / 1 X , 
t * 5 A 3 LNfidXXXXXX.XXX,  1 ',/lX, ’SCO 

2100  FORMAT  ( IH  , t Y , » N * , ?X  , ' C E L T A • , A X , ' A LPI  A » , 8 X , ’ A PF  ' , 8 X , ' FF  F G > , 
l iax,>T',10X,'SMR',8X,’EPIN»,//'lX,13,tX,I4,tP6F12.4//) 

22KK  F ORP  A T ( 1 H , L>  X / ' £ I G E N V A t.  L F vS  ' , A X , ' (I  MT  I AL  IJ-S  fl  E SP -C  P T I P A L 1 , 

1 • U-S  PESP’.)  * ,5X,  »RX;<<n  • , /1X#SSX,  » C C U P l ED* , 10X,  ’ i.NCOL  FIFT  ’ #//) 
2 3 OK  FORMAT  (lb  , 1 X , 1 FF'l  2 . 4 , 2 (CX  / 1 F E i ? 4 ) ) 

2427,  f OP.F  A T ( 1 H , 1 F_  N 7 E R l:F  7C  0 CLRVfE  TC  PE  GpMF  P A T H C » , / t X , 

1 >0  = NO  CURVE'  ,/tX,  d=ll'S  GF  AC  Tf  X T ' , / 1 X , ’ 2uJ  A CCP  t RELAXATION*  , 

2 /tX,'3=30R  PETf-CC? ,/tX,  '4»CATA  PILE  FRCP  CXSX'fStX# 

:j  '«jsy jKiMiij*  pse  e n u f o for  sxep',//ix, 

4 »()()()()()<)()()  ') 

2500  F0R?'AT(1H  , 'NIPPER  CF  POINTS  TC  OF  GE  NEPATFC  ' , / 1 X , ' C )') 

2650  FORKATCtH  » ’ FN  T FR  UP  TC  8 VAPIAEUS  FOR  PLOTTING '#/ IX# 

1 '()()()  C ) ( ) ( ) ( ) C ) ' ) 

2700  FORPAT  ( 1 H , 'ENTER  STARTING  ANC  FIMSF  POINTS  FCP  PLOT*, /IX, 

I « NST  APT  iNFIN’#/lX,  » ( )(  ) ») 

2750  FORMAT (IP  , 'ENTER  1 FOR  AUTf SCALE « / /l X# » ( )») 

2620  FnRPATUH  ,'FMER  CCNSCLE  DEVICE  (6  = 7Ek,  RsCCXSCCPE  ) ’ , i \ X , 

1 '<  )') 

282e  FORPATdP  /'ENTER  l TO  CONTINUE  F!  CT  T TNG  • , / 1 X , « ( ?') 

2852  FORP  A 1 d H #'FNTFR  I TO  GENERATE  N E SET  CF  C L R V F 3 ' / / t X / ' ( )' 

292e  FORP  A T ( 1 H / ' F.M6R  i 70  READ  N EX  E T-GENVAl  UP  ?t  *T  A ' # / ! x#  » ( )•) 


non  non 


29 


SUBROUTINE  U-2<T,K^P,KC,en;,EVIN) 

COMVCN  XO»*ii2) 

DIMENSION  glG(N)#HjC(N),A(32)fEI62(32) 

* 

ASK  VALUE  CF  Mj 

WPlTFCS.SPPfl) 

PE  AC  <5,  }«•«??  ) U . 

CONFUTE  CONVERGENCE  CURVE 

X(I#1)3F}41N 

CO  6P  J ^ t » N. 

A(J)b£1G(J)*KC(J)**:? 

X(l,n»X(I,|)  + ACJ) 

5-2  CONTINUE 

DO  120  Ka2,NP 
:<f  I,K)s6MIN 
CC  75  J-l,\ 

A(J)*A(J)*ETGPfJ) 

X(!,K)tV(I#K)+A(J) 

75  CCMInUE 
10S  CONTINUE 
RETURN 

\ll°.  r'OF^ATt  \ :<,FlC5,e) 

20*V»  FPhV  A T C t H , ’ENTER  NU  VALUE  FCR  INC  6PA0TFNT  NF7NC.C  » , /I  X 
1 ' C ) ») 


on  n 


JHIS  P Ml* 

|W>M  OOrX 


IS  BEST  QUALITY  FRACtW*®^ 
IfuKhlSHED  IODD0  — — 


SUfi^CLl  L.Nt  J A L C E I ( I , N / • (s  r , r L C , E I L , (<  X X , C ?'  I N J 
CORNCN  H#,5!ii) 


DIHt. N SIGN  fc  I C I N > , h L C I .S } , A ( 3 2 ) ,c  1 0 2 ( 3 2 ) , fi  X X ( N ) 
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ASK  VALUE  OP  ticiTA  ‘ 

WHI  TE  I6,2Ut;idJ 
Kt  A 2 ( 6 / i W*32  J ft 

L ASK  NC.  SAf^LtS  U 6 T 'A  e F N IfEMTlCN  STEPS 
C 

ART  It  I6,2L’OD) 

KEAOIO/  u/bj  ns 

c 

C CONFUTE  LGiiV|£HbtNL£  CURVE 
C ‘ 

xu#i)*tniK 
oc  se  jm,n 

eiG2(J)*(l. -£*£!€(. J)/KXX(l))**2 
AtJ)*LlOI  j)*KttJ)**2 
xU/ 1 3 * x o ; 

0 4 C 0 N f { Mi  fc 
NCOUM  «U 
00  iiti  Sek/NP 

nCCUM  aNCOOlWU 

Xf  UCOUN  J ,L1  *NS)  GO  TC  fllfl 

NLGUMsU  ‘ :' 

Xl  I , K) JENIN 
D'j  7’J  J‘1»N 

At J)«A( J)*6  IG2CJ) 

X ( I / K ) * X l i / K ) * A ( 0 ) 

75  CONTINUE 
GO  1C  ItJ0 

SS  XU#i<)*X(l,KMJ 
C 0 N i 1 Mi  E 
KtrOjlN 

1800  rORNA f C IXfFiV'i  ) 
uae  FCRX4HlXfl2) 

ataC  rORPA]i.lP  , 'EMER  & f f A VALLE  PCS  J.4CC0I  RELAXATION  ?£fKC, 
\ ’ ( ) ' ) 

2200  FORK/UUh  ,'fc.MeR  NC.  SAMPLE  STEFS  F E H ITERATION  STEP  TCR', 
1 • JACCUI  RELAXATION', /IX/ ' ( )')' 

fcND ‘ ... 


r*  o o r-*.  r>  /->  non  r*  o o o o r>  o o r> 


TQR«ST4UiLmFBACXlC^ 

IBIS  MQ*  IS  B*SI  ^ ■"«  ^ 


'.V- 


SvJqpcut [ye  scRn»N,KP»H,«xx«e»'iN) 

CCFNCN  X(S,?12) 
rivt  xani!  wu 


ASK  VALLE  CP  PE  T A 


WRITE(fi,2023) 

REAC  (6*  * '*££  ) EETA 

ASK  NO.  CF  WEIGHTS  LFCATEO  PER  TTFFATTCN  S7FP 


K31TET6,?m) 
REAC(5,  i » T f* ) -M> 


ASK  NC . CF  SAMPLES  BETWEEN  ITERATION  STEFS 


hSIT6£C,22C!P) 
REAC(5,tl'!®)  NS 


LOAF  WORK  TNG  ARRAY 


CO  133  J=l/M 
hSK.DahCJJ 
100  CONTI Mi*{ 


CONFUTE  CONVERGENCE  CURVE 

:<  ( 1*1)  a A f»  S F (H  > I R X x , E N I N » N ) 

NCCUNTs? 

N S T s 1 

00  5 00  Ms2,NP 

CARRY  01 T SCR  NETKC  WITH  PARTIAL  ITERATION  STEP3 

NCOUN  T sNCCLiNT  H 
IF(NCCUNT.LT.NS)  GO  TO  400 
NCOlNT  *0  ■ 

MJF*S! 

252  IF(NST.OT.N)  :NST»t 

HV:(NST}5n%-BETA)^FK(NST)*FXY(l)/PE7A 
CP  30V  J s 1 , n 

IF  C E Q , N 3 T ) GC  TC  330 

K»1  A$3(  J-NSD-H 

Hfc (N81 ) *PF(NS7)-HN ( J)*RXX (K) 

320  CONTINUE 

HW(NST)sHN(NST)*EETA/RXX(l) 

NST»NSTM 

NUPsNliP-M 

IF(NLP.LT.NW)  GC  TO  253 
X (I / K ) * A M 3 E ( h * , R X X , E M I N , N ) 

GO  TC  500 

403  X(I#V)aX(I»M-t) 

530  CGN T I Nil ? 

RFTL'FN 


KS  PAGE  IS  BSST  QUALinnaciLCAfla 

»i-mTi  Trt  nDC — ^ * 


1002  FORMAT  (IX, ft  0.2) 
t 100  FCRvAT(tX/  I?) 

2302  F0R'VAT(1H  , ' E M E « BETA  FOR  SCR  A l C C R I THK  » , / i X , • ( ) M 

21PC  FORKATtlH  , 'ENTER  NC,  fcE  IGH  S 1 PC  A TEC  HER  ITER/.HCN  STEF  PCF», 
1 ' SCR  ALCCP  m-K  ' ,/iX,  • ( ) • ) 

2202  FPRKATUH  / 'ENTER  NC,  SAMPLE  STEPS  PER  ITERATION  STEP  F C R » # 1 
1 » SCR  ALGORITHM  »'  ,/lX,  ' ( ) »'  ) 

END 


F U r f T 1 C N A K S E CM*  / F X > # H K I N / N ) 
DIMENSION  Kta(A),RXX(N) 
ASETsFMIN 
CO  220  Jsl/M 

A *3,3 

DC  J20  K=1,N 
L a I A33 ( ) 4 1 
A j j*  ->  ;•>  X X ( L ) * P * ( K ) 

130  CONTTNU? 

A ; T s A s T F A * I*  X ( J ) 

200  C 0 O' T 1 1: U r 
A w 3 i:  -s  A 0 F T 
R i:  7i!  R N 
tNO 


r>  o o o o o 


mis  pin  is  bssx  quality  praczi&asu 

/BOM  (SOFT  rURMISHED  TO  DDC 


SUBROUTINE  0D,Y7A(T,Np) 

common  xcn,5i?i 
DIMENSION  V(.m/Z(5t2) 

ASK  FOR  CI3K  FILE  ASSIGNMENT 

WRITEIS, 28821) 

PAUSE 

DFFTNF  FILE  2 <-£12  / 23  / U*  INDEX) 

ASK  WHICH  C 0 ’J  N rf  CATA  FILE  xg  CEE  '(FED 


C 

C HEAD 


1000 

223:? 

?1?3 


WRITE  (5, 2 ISA!) 

REARCfi,  U-ifl)  .ML* 

! DATA  c{LF. 

DO  lew  JlifNP 

RE»0(r>*  I)  (Y(X)/Ks]  ,11) 
21.1)  ay  (NO 
CONTINUE 
END  FILE  2 
DC  J=l,NP 

X ( I , J ) s ? ( J ) 

CONTINUE 

RETURN 

FORMAT!  IX, It) 


FORMAT  US  , ' ASSIGN  CATA  FILE  TC  EL  PLOTTED  fC  UNIT  2,  EX  AMP 
/1X»  'SA3  DK8iXXXXXX.XXX/2'*/lX«I.<iC0>) 
f OMM  A T ( 1 h t 'ENTER  DATA  RECORD  TC  8S  PLOTTED  5 ; /’lX  , 1 i *SISML  * 
»2*CP1IVAL  SIGNAL  ESTIMATE1, /t*» »2«LMS  SIGNAL  'ESTIMATE1 #/!X 
UvJACCOI  SIGNAL  E ST  IM  A T E ' t / 1 X , * 5 ■ S OR  SIGNAL  ESTIMATE*  » / J X , 
1 C = I.  E V I N SON  SIGNAL  ESTIMATE »,/! X ,» 7«MSE  FOR  CPTIMAI.  FILTER- 
FCF  L MS  METHCC* #/lXf »9»M8E  F OF  JACOPI  METHOD  ’ » /IX , 
•I0»M$E  FOR  SC F MET-HGD>#/JX,lll*MSS  FCF  LEV TNSCN  ALGORITHM* 

■ < \ t « 


PC  ) ») 
END 


Oi  rv 


^ ■ ■ 

299 

SUGRCUT1NE  PLOT  IT  (NF/NVrNS,FVAF,MiTAiU,NFIN,ICEV) 
iNTtC- tr(  PVAH 
CCMWCN  X(3,D12) 

Di,NtNSICNXMAXCe),XPlNC<J),XNfcAN'e),PVA{Uf.V) 

call  chcn ruisp/ icsf/ iflus» KOLLf lr.ev) 

c ... 

SCALE  DATA 

N3  f *NSTART->1 
DO  JZ  J = 1(.‘NV 
XP£A,N{  jj«irt,0 
X A X ( .J  ) 

DO  1 i!  I » N 3 1 a i<  1 , N F I N 

X P e A .n  y,  J ) s X F h A M j 3 ♦ x l J , I ) / f L c A T ( N F J N - N O T A R T ) 
lk.  AMAX(j)*AFAXiai'AXlJJ/MJ,l)) 

XOINU)sXrAXC  j) 

00  15  I«Nai AH!, Nr  IN 
15  XNIIUJ)*AMN1  ix.MN(J)  ,X  tJt  1)  ) 

IF(XFAXCJ))  2<S,2a, 32 
2k)  xHAX(J)iXPlNlJ)  ‘ 

LU)  25  I*NSrAKi,NFlN 
25  X^AX(J)=*Af'AXliXf'AX(J)/X(J/I)) 

J55  CONTINUE  

K»l' 

DO  52  J-A/NV 

IFi ( P VAH ( J ) *G T .0 ) . ANC , ( P V AH  U ) ,l.t  A-  V ) ) KffpVAK  ( J J 
52  CON  f 1 »“•  U h 

XPAX 1«XPAX (K) 

UC  2?  JM^NV  ' 
lap  VAR  U) 

ifcu.lt.d.ch.u.gt.nv))  -GC  TC  27 
XMAXI«APAXl(X.,;AXl,XCAXCi))'‘ 

Xi!INl«APlMUMM#XMN(I)  j 
27  CONTINUE 

lF(NS.Eii,«)  GC  TO  5 2 
Y 1 N 1 3 ( X '<  A X 1 - X !*  I N I ) / 2 « + X tv  I N I 
40  IFCAESUFAXI-YINT  j,NE.li.B)  GC  TC  A 5 
YSJl, 

GO  TC  5 J 

AS  YS*3l5,/Aa.S(Xf'AXI-YINT) 

52  X3*022 ,/FLGA (INF IN-NSTAHT) 

IFCNS.Nt.tf)  GC  1C  65 
CALI-  TF.NPLT  (0rU#tf,e,Uf  ICEV) 

KHI7EC6, 25(00) 

cu  6e  ‘ j«i,'nv  ' 

lapVAR(J) 

lF(U.Ll,l).OH.a.GTtNV))  GC  TC  6U 
WHI  U C6,24SUJ)  'iJXPAX  (I),XHmi),XPEAN(I) 

60  CONTINUE  ' .... 

MRltE(6?2&00) 

REAUC6,  UKg)  Y3 
YSs45./Y3 
VUU  f E (6/26fiM) 

REALCft/llUfJ  TINT 
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65  CALL  UKPLT  ,0,0,  ICEV  J 

CALL  TEKPLTi2f*%,  Itf  /C5li,  i Id  r ICE  7) 

CALL  TfcKR-T  t?.f5K,642#8::K,6  4id,  ICEV) 

CALL  TtXFL T (2, 5H,  10,50,640/ ICEV) 

CALL  TEKRLT  12, 6 5K, 1C, 850,640,  ICfcVJ 
CALL  TEKFLT  C2  # £ k£ , 325/  0814, 325/  ICEV  ) 

00  lee  j *i / i 5 
1Y»4S  .*(  ib-j)-ue, 

CALL  TEKPLT  12,60, IY,OW,  i Y,  ICtV) 

CALL  TtXPLT  12, MK , IY/8BL/ IY, <UEV) 
X?sYIMt(4Si/UJ*PLCAfi8-J) 

1 Y * 1 7 • 6 

CALL  7 EKPI.  17,8,2,  ICEV) 

1 U t*  WHIfEUG£'V,$im)  1FIU3,XP 

co  tsi 

ixabrj*  j+aa 

call  UKFLT12,  IX,  1U,  IX,2t’,iCtV) 

CALL  rEKPL  142/  1 X , 3 2 tt  , IX  , 3 J«i  / iU.fc  V ) 

L be  CALL  |£KPLTi2/lX/6Je,IX,64!i/ IOEV) 

00  800  M 1,NV 

J’PVAHCiN)' 

lF(tJ,L'f,l),Oli.U.aT.NV))  C 0 TC  880 
m^UU/NSTAhl  J-Y  IN  n*7S* 325, 

CALL  TEKFLT 13,5,' 1YY, 5, IYY,lUEV) 

MKl'iE(IDEV#27«ej  1FLU5,J 

cc  rtfcC'  i*,\sT,AFiN 

K1Y  i s (X  ( J,  1-D-r  I NT  JAYS  >325  , 

KlYfiaUiJ  / I)-TlM)*Y3  + 325. 

I F ( A £ fi ( ; \ T Y 1 j . L T , J „ E ♦ 4 ) F IYl * 3 , E ♦ 4 
1 F C AES  (K n 2 ) 7 . 3 s E ♦ 4 ) F I Y2 *3  ,.E  *4 

IY1-1M  IF  iU)  " ' 

l y 2 - 1 f»  f iniY2) 

K«I-NST‘" 

iXl*(MXS)+90 

lX2*(Ktl)*X5F3e 

CALL  TSHfLT  12  / 1X1 , mr  1X2/172,  ltEV) 

CONTINUE  ' '• 

FORMAT (IX/ l») 
format  iix/Fie«gj 
FORMAT  1X2/  l PE  10^3) 

FORMAT  (/IX,  'VAN  IAHLE  *,SX,  • Y N A X * , { 1*,  »YM1N'/11X/  1 M E A N * / / ) 
FORK/*  TC  «X  / 15,-318X,  lPEig.3)) 

FORMA  IU/1X/ 'fcliUli  Y-AXIS  SCALE ’ /lXf ' ( )s) 

26K)»  F9KMM(//iX/ ’cMKH  X-AXIS  F QS  II  1CN  * / U / * ( )') 

27W2  FORMAT  CA2;i2)“  ' 

2200  FORMAT (///) 

RETURN 

tND " 


uevi 
uue 
1 1 v e 

2014? 
2 400 
2500 
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D.4  Program  for  Computation  of  DFT's  of  Signal  Estimate  Records 

This  program  was  used  for  the  computation  and  plotting  of 
the  DFT  of  the  signal  and  signal  estimate  records  as  described 
in  Section  6.5.  The  last  20  samples  of  the  data  record  from  the 
results  of  the  simulation  program  are  repeated  51  times  for  a 
total  of  1020  samples.  The  1024  point  DFT  is  then  taken  of  the 
resulting  sequence.  This  procedure  was  followed  since  the  FFT 
program  is  a radix-2  algorithm  and  it  was  desired  to  reduce  the 
effect  of  windowing  present  in  the  output.  The  program  was 
written  in  Fortran  IV  and  run  on  a PDP  11/45  computing  system  in 
an  interactive  fashion. 

I 

D.4.1  Program  Structure 

I 

The  structure  of  the  program  for  computing  and  plotting 
the  magnitude  of  the  DFT  is  illustrated  in  Figure  D.4.  The 
program  segments  are  described  on  the  following  pages. 
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1.  SPECT  — Main  Program-  Calls  program  FFT  and  PLOTFT. 

INPUTS : 

IDEV  — Logical  unit  number  of  graphics 

terminal . 

NP  — Number  of  records  in  data  file. 

NR  — Identifier  for  the  particular 

variable  desired. 

Data  file  name  assigned  through  monitor  command. 

OUTPUTS : 

2 — Array  containing  magnitude  of  DFT 

2.  FFT  — Computes  the  DFT  using  the  FFT  algorithm  by  Brenner. 

INPUTS: 

DATA  — Array  containing  real  and  imagin- 

ary parts  of  data  stored  in  al- 
ternating locations. 

NN  — Number  of  DFT  points  desired. 

ISIGN  — -1  for  transform,  +1  for  inverse 

transform. 

OUTPUTS : 

DATA  — Array  containing  real  and  imagin- 

ary parts  of  result  stored  in 
alternating  locations. 

3.  PLOTFT  — Plots  magnitude  of  DFT,  calls  subroutines  TEKPLT 

and  CRCNTL . 

INPUTS : 

X 


NP 


Data  to  be  plotted. 

Number  of  points  to  be  plotted. 


3 


N V — 
NS  — 


NS TART 


NFIN  — 


I DEV  — 


NV=1. 

NS=0  for  manual  scaling,  NS=1  for 
autoscaling. 

Location  in  array  X at  which  plot 
is  to  begin. 

Location  in  array  X at  which  plot 
is  to  end. 

Logical  unit  number  of  graphics 
device . 


OUTPUTS : 

Plot  of  the  Array  X. 

TEKPLT  — System  subroutine  for  vector  graphics.  See  sec- 
tion D.3.2  for  description  of  inputs  and  outputs. 
System  subroutine  for  generating  carriage  control 
characters.  See  Section  D.3.2  for  description  of 
inputs  and  outputs. 


CRCNTL 


_ TS  best  P^CTI 

JBIS  IS  BEb  J0  UDC 

{ggOOiF*  FUK»I^  w 


PROGRAM  PGR  SPECTRAL  ANALYSIS  CF  ALE  CATA 
3Uh  6 /'22/7S 

0 1 M t N a 1 U N X(2«48),REC(U)fH2Q)F2(lQ24} 

ASX  KEYBOARD  CCVUE  NUMBER 

W R 1 1 1 Cft/20001 
KtAC  <6,  IdHR  J iL'EV 

DEFINE  l'  4 1 ;*  riLt  ON  Ur.  11  2 

20  WRITE  15/ 2 100) 

PAliSc' 

DEFINE  PILE  21265,22/U,  1NCXJ 

READ  NUPCtK  CF  RECORDS  IN  FILE 

'.IX  r ! L C 0 > 22'uiu  ) 
i< c A L ( i / 1 lttc'J  N F 

READ  DA]A  FILL  DESIRES 

WH  J I £16/2000) 

K t A 0 i ft  / 10UK  ) ‘Mi 

READ  D A t A 

dc  ne  1*1,20 

ft  - ,\  F - 2 3 * I 

REAGI2‘«)  CN£C(J),Jai,li) 

Yl  I ) *I<EC  (Nft  J - • * 

100  CONTINUE 
END 'FILE  2 

idERC  ARRAY 

CO  Std  I ■*1/2048 
X U ) * 0 . 8 
50  CONI  1MJE 

LOAD  ARRAY  FOR  FF1  PROCESSING 

DO  200  131,31 
DC  150  Js  1,20 

ft  s A 0 A ( I - 1 j ♦ 2 * J - 1 
X U ) * Y f J J * ’ 

155!  CCNtlNUk  “ 

200  CONTINUE 

COMPUTE  rCURlEw  T ft  A N S F C R H 
LA(.L  FOUR  i (X,  U24/-1) 


i r*s»  C'.%  cx  » r*i  r*.i  r*i  rr>  r,  »rnn 


MirifiiiA'i: 


THIS  P 


•SOM  OOPY  f UhAISHED  XO  DOQ 


TAKE  MALM  f ODE  Of  S c S U L T 


DC  2 5 0 .13  1,1224 
K*2*  1*1 

Z(I)3S<iRTlXlM**2tX(Ml)**2) 
250  CONTINUE  * " - 

PICT  KESCUT3 

ASK  FCK  A U T C SCALE 

3?2  WRITE  CM  24X1!) 

H t A L l i t 1 «J  A ? ) A c 

PLCf  SPECTRUM 


CALL  FICTM  (2,  1024,  l ,NS,  1024,  I DEV) 

WHITE  t6,25u«)  ....... 

KEAC  (6, 100fc J NSl 
iFtNSf  .tC.l)  .tC  " T C 500 
*iK:r'c(e,2o0wj" 

PLaC  1 1,  ifc'L't  J f.  I 
1FCNS  r.LL.toJ  LC  '■  10  20 
Is UN. HA  Ills,  li)' 
r*  L H H A f l iX,  13  j 

POWPAfUP  ,'LMSP  KC:  DEVICE  C e » 7 £ ^ , 9«DEKSCCF  E ) • , / 1 X , • C )* 
F C fi  v A T C l h , ’ASSIGN  DATA  FILE  tC  17-37  2*3 


l <i  'J  i 

1 : 0 2 

2 '.1  / z 


i J 4 0 


2 502 
2 022 
2 422 


FurtO  ruii  , 1 ML  f'  L‘  EH  Cf  RECORDS  If  F ILS * , / 1 X.  » < )») 

f UPrAU  il-  , 1 ENTER  DATA  RECORD  H E S IK L:C  ‘ , / 1 X , * i *S  1 GN  AL  » , } 1 X , 
t * .2  • C F T X M A l * 3 It  M L E 3 UP  Ai  L * , / 1 X , * 3 *LMS  S IGM  AL  ' E £ 7 IF  A f E « , / 1 
4 *4  = jACOtil  SILaAL  tSTIf  ATh'i/lX, 'CaSCR  SIGNAL  E 8 7 I N A TE ' , / 1 X 

3 * 6 * l t v i s d c i\  Signal  est  uau»;//ix,  *<  ) n~ 

FOnMMUh  f'bMbli  1 TO  PLOT  AGAIN', /iX,»(  )?} 

FQRHArtlH  ,’BMEH  1 to  EXIT  FRCP  PGK',/1X,T<  )»I 
FORMAT  C 1 H RENTER  1 FOR  AL  I C * SC  AL  t » , / 1 X , » X ' j » j * * 


• Am' 


2 0 B R C l T i hit  r'LCTF7(X#NP,T\V,N2,!<S7ART,Nf:IN,ICEV) 
LUMENS  ICIN'  Xi'l'i 

CALL  CHtNTlUSF,  1C5F,  IFLUSr  ICOLlf  1C6V) 

SCALfc  LATA 


NST«NSTAHT-U 

XT,£AN«0.B‘ 

XFAXs0.0 

DC  \'<d  J*NSTAR1 ,NFIN 
X!*£AMxrfcAN  + XilJ/FLCAT{NFIN-NS!ART) 
1 'L  X ,v  A X * A i*  A X 1 C X y A X ,'  X ( I)  ) 

X i“  I X s X A X * 

L 0 It  lsNS7ARI,AFIN 
15  XT'  X T ' A i*  1 .%  U X1UN  X U ) ) 

XhlX^AAJ  '-'a,  20,30 ’ 

20  X^AXiNtfiN 

UU  25  I a fs  S T A R 1 , N F I N 

25  Xf'A/UAi-IAX'i  (XMAX,X  U)  ) 

32  CONl  JNOfc  * 

Xh*AXIsx'“AX 
X p:  i N .t  sXMTi 

IF(NS.£4i.itf)  OC  1C  50 
Y I N f ■ UMAX  l-XFJM)/2«*Xt'INl 
Aw  IF  ( Atfs  (Xf-'AXl- » IS  1 ) .NE.2.0)  GO  TC  45 
Y3« 1 ' 

GO  1C  52 

At  YSOl!;./ AOS  IXi*  AX  X-YINT) 

50  CA1  \,sir  IN -X START) 

iF(*3  .Nfc.S)  &C  tC  ft 5 
call  ui<FLii0>e;gfe,0ficEv) 

5R 1 1 1 lc / 23fi0 ) 

y*Hl  ffc  (6/2402)  1',  X P A X , XfU  N , X Kfc  AN 

WRIi£(6f2500i 

iifcAJU/  i 120  ) tS 

Ybs^s  ./  ?3  ' 1 ’* 

flRI 7 £ (6,26001 
REAOU,  11001  TINT 

es  Call  ukpl  f <0',0,C,s,0,  ilev) 

CALL  ft-KFLT  1.2/ 50  r 10,650,  V£  i 3CEV) 
call  7fcKF 07(2, 890,10, 250, 640, itfcV) 

00  1E0  J«i  , i 5 
iY  = 45.A  u&»j)nu. 

CALL  UNFIT  Uf640,  lY/SSB,  IY,  iU£V) 
XPsTUftcAs./Yg  j*FLCA  r(8-j) 

IYalY-6 

CALL  UKFLT  (3,695, XY,0,0,ICfcV) 

100  wRnE(XO£V,200fc>  IFIUS,XP 

1YY*XXCN31AKTJ-YINT)*Y2*325, 

CALL  UKPI-7  (3,'S,  1YY,5,  IVY,  1CLV) 
NRITEUUIr.V, 27VSB1  1FLU3,J 
00  «0(S  i8r<ST, NF  iN 
P I Y 1 8 ( X ( I ~ 1 ) *•  Y j N T 1 * Y 3 *■  3 2 5 , 

KI  r2»  tX  ( X A- YIM)*YS‘»J2e» 
iF(AK£(RIYl  J .fit  ,3.fe»  4)  RXYlaJ.E** 


V ^ 
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lFlAeS(RlY2J.-l.T,3.£M)  IUY2»J,fc*4 

i u * i n f i r m ) 

I Y 2 * I N T IH  1 v ii  > 

K«l-N3r 
lXt=lkAX3)+5W 
lX2«V/<  + I J^XSttK 

call  TtkFLT  ^/'lXUIYl,  1X2, IY2/  ICfeV  J 

839  CONTINUt 

I JO  2 FCRMAIUX/I5) 

1 i kJ 3 FORMAT  uX/Fiii.ej 

2 a 03  forma  T i.  42/ 

2 330  FORMA  U/iX,  • VAMAFui:  ' ,5X,  HMAX' , UX  / » YF1.N’,  1 iXr 
2400  FCHMJTi  IX/  10,3  t 8,X  , * rF.12.3)  ) 

2530  FORMAT  l / / 1 X / ’ L N f e R Y - AX  I S 5C  A Lf  « / i X , » < 

2300  FORMAT  W/lX,'ttN  IfcK  X-AXIS  F C $ 17  i C M / i X , » ( 

2732  FORMAT i A 2 / 12)'  " ' 

2232  FORMAT  (///) 

Kt  TURN 
6N0' 


,SBSS, 

jmSPAGETS  II)  UDC 


1 


'MEAN'//) 
) *) 

‘ 'N 


c 

C f H I o IS  fFt  PGUR1EH  TMNSFCRP  F R QF  P i 7 BY  BRENNER, 

C 

C DATA  IS  STURtC.  IN  1FE  2*NN  X 1 ARRAY  C A T A ( I ) BY 
C ALTERNATING  KfcAl.  ANU  IMAGINARY  FARTS  RITh  CUD 
C SLrioCHIPTtU  ElEMtMS  CCN  1A  I N 1NG  'Fit  AL  FART  ANO 
C TFc  FOlLGrtlNG  EVfcN  SEUSGRIFTEC  kLEPEM  CONTAINING 
C TFE  CGHRtSPGNCiNL  l, ''AGINARY  FART,  NN  IS  T F £ 

L U5NUTH  *C.U  TFfc  TRANSFORM  Ain  C FIST  £ £ A INTEGER 
t RCWEN  OF  2.  I31GN~SFCULC  EE  cCGAL  K - i f’Cfi  A 
C FCitWAHO  TKAi-iSrUNF  ARC  ♦ l FCR  AN  I N V t R S f.  T H AN  SF  C r F . 

C > F£  H £ 3 1 L f A !\  G T H A f\  S r i; .« \"  IS  ;<  t T l R N £.  C IN  T M.  A l<  H A Y 

c u a r a ( i j siUfi-c  ;n  TPt  safe  fasmun  as  describes  above, 

C ■ 

3UQHCL  T i Mi  F'JLKl  l C A T A , R \ , 15U-N) 

dimension' c a i a i i j 

|'a2“NN 

CU  & I*1,N,2 
t F ( 1 ~ . "u  1/2  ,2 

1 rF>sJli«*UATAt  J) 

T t ?'  F l 3 IT  A T A i 3 + 1J 
G A t A l J ) 8 G A 1 A 1 1 ) 

OATa;.)*1  )s04  JAiltl) 

BAT  A n I i * T'cM>;n 
DATflUi  U8IEivPI 

2 RsR/2 

3 IT  U-P  3 5 , S , <1 
A 

M*P/2 

IFCM-2)  5,3,3 

6 jS.J  + F 
KMAX«2 

5 lF(MFAX-N) 

7 I3Tfc?s2*,'!f'AX 
BU  9 Fsi , UMAX,  2 

lPETA»-1«l41&S28b35*F.l.i:AlU3XGMlP-)))/FLCAT(?’r,.4X) 
WH*COSlYr.ti  A) ’ 

W I « S IN  ( IT  tl  A) 

DO  d i«P!,N,  iSTSP 
' ’Jal  + FMAX 

I EMFR'InKAGAT  A (.  J)-vU*CA  I A C.J+1  ) 
l£FT;I»WK*CAiA(siHJ*T\I*BAiA(jj 
CATa(J)*Da]  A (j  Jr  TEN  PR 
CATMOMJtCAlAil  + JWbf'Pl 
CATA<i)»DATA(lj-»TEPPH 
CATAU  H j *BAT  A ( 1 + 1 )*T»:PP  I 
B CONTINUE  ' " " 

9 CONTINUE 
MMAXsISltP 
GO  1C  6~  ’ 

1M  HE  TURN 
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estimation  and  if  the  statistics  of  the  signal  and  noise  are  known,  the  optimal 
unit  sample  response  can  be  foun  ‘ ' 
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20.  equations.  The  resulting  processor  is,  then,  an  example  of  the  classical 
Wiener  filter. 

When  the  statistics  of  the  signal  and  noise  are  not  known  a priori,  the 
unit  sample  response  of  the  Wiener  filter  cannot  be  determined  and  some 
method  for  approximating  the  optimal  filter  is  required.  One  approach  to 
this  problem  is  to  use  an  adaptive  processor  which  attempts  to  "learn"  the 
^^characteristics  of  the  signal  and  the  noise  and  to  use  this  information  to 
\qbtain  an  approximation  to  the  optimal  filter. 

In  this  report,  the  performance  characteristics  of  the  LMS  Gradient  Al- 
gorithm, two  Adaptive  Fixed-Point  Iteration  Algorithms  and  of  a non-iterative 
method  based  on  Levinson's  Algorithm  are  considered  for  the  case  where  an 
adaptive  algorithm  is  used  to  determine  the  unit  sample  response  for  a system 
which  attempts  to  discriminate  between  the  signal  and  noise  processes  on  the 
basis  of  bandwidth.  Such  a system  is  often  referred  to  as  a Spectral  Line 
Enhancer.  Theoretical  bounds  on  the  mean-square  error  as  a function  of  the 
time  index  n are  derived  for  each  of  the  three  iterative  methods.  A 
comparison  of  these  bounds  is  made  for  the  case  where  the  input  data  to  the 
Spectral  Line  Enhancer  is  composed  of  a single  sinusoid  of  random  phase  in 
the  presence  of  an  additive  autoregressive  noise  sequence.  The  results  of 
extensive  computer  simulations  of  the  adaptive  algorithms  considered  are  used 
to  determine  the  usefulness  of  the  theoretical  bounds  and  to  make  comparisons 
of  the  performance  of  the  four  methods. 
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